Software Engineer (Pipeline Services) / Data Platform

求人概要 / Role and Responsibility
■ About the department
LINE Corp is a provider of its renowned messaging service and a wide range of other services, including finance, content and AI, to our hundreds of millions of global users. These services generate large amounts of data every day, resulting in LINE accumulating over 250 petabytes of data. As stated in "LINE STYLE", which defines the LINE practice and mindset, we take an “always data-driven” approach and leverage data as an asset across the company.
The Data Platform Department aims to democratize data and develop robust machine learning infrastructure. To this end, the department offers a platform that can leverage an enormous amount of data efficiently to drive service growth while also helping engineers, service planners, and marketers to capitalize on it.

■ About the job
As a Software Engineer for Pipeline Services, you will be developing a large scale data pipeline commonly used across the company that connects LINE's various services and Information Universe (IU), our proprietary data platform for data collection and analysis.

We build two types of data pipeline: one using Apache Flink for data streaming and the other using Django and Spark / Sqoop for snapshot batch data in database. The data pipelines are deployed on Kubernetes having as many as 1,000+ nodes and automated based on ArgoCD. You will play a key role in providing an architecture that allows flexible data design and advanced scale-out strategy in order to accommodate a variety of service and business needs, thereby supporting LINE services' sustainable growth in a data-driven manner.

・Dataflow capable of high availability and high throughput
・Incremental processing for large scale data
・Consistent CI / CD pipeline throughout from development to release
・Protocol design enabling flexible logging

This is a challenging position as you will be expected to understand and solve complex problems involving over petabyte-scale data. You will be provided with many opportunities and support to develop your skills and abilities to tackle organizational challenges through regular meetup events for data engineers and OJT.

■ Tools/development environments
・Streaming processing: Flink / Fluentd / Kafka
・Batch processing: Spark / Sqoop / Airflow
・Distributed environment: Kubernetes / YARN
・CI / CD: ArgoCD / DroneCI / Jenkins
・Operation and monitoring: Ansible / Grafana / Prometheus / Promgen
・Distributed storage: HDFS / Elasticsearch
・Query engine: Hive / Trino / Spark
・BI tools: Tableau / yanagishima / OASIS (internal BI tool)
・Development environment: IntelliJ / Github
・Language / Framework: Java / Kotlin / Scala / Python

■ Reference information
Flink@Data Platform - Ingestion Pipeline Redesign and Auto-scaling
応募資格 / Qualifications
■ Required experience and skills
・Experience developing/operating/troubleshooting a system in Linux environment
・Understanding of computer science fundamentals including data structure, algorithms, and computational analysis

■ Preferred experience and skills
・Experience developing streaming data pipelines using Flink / Spark
・Experience developing batch pipelines using Airflow, Luigi, Azkaban, Digdag, etc.
・Exposure to large scale distributed Big Data platforms such as Hadoop, AWS EMR, Cloud Dataproc, and machine learning pipelines
・Ability to build a system using container technologies such as Kubernetes
・Experience developing applications based on a message broker such as Kafka, AWS Kinesis, Cloud Pub / Sub
・Experience automating / optimizing operations using orchestration or monitoring tools
・Active contributions to open source projects

■ Ideal candidate
・Capable of proactively finding and solving problems
・Interested in distributed systems and data
・Have a willingness and desire to learn new technologies and be eager to take on new challenges
・Capable of capturing diverse user needs
・Able to involve and coordinate with other teams when necessary
・Able to tackle and solve difficult / complex issues
勤務地 / Location
〒160-0004 東京都新宿区四谷1-6-1四谷タワー23F
雇用形態 / Employment type
勤務時間 / Working hours
専門業務型裁量労働制(1日の労働時間に関わらず1日9.5時間労働したものとみなします。)、フレックスタイム制(コアタイム 11:00~16:00)、10:00~18:30(実働7時間30分)のいずれか適用 ※面接後に決定
待遇・福利厚生 / Benefits
■休日 / 休暇

 - 年俸の12分の1を毎月支給。
 - 別途、インセンティブプラン有(※1)
・諸手当:交通費支給(会社規定による)、LINE Pay Card Benefit Plan(※2)




 - 四谷オフィス、南新宿オフィス、大崎オフィス

※当社はオフィス勤務と在宅勤務を組み合わせた、より効率的に高いパフォーマンスを発揮し続ける新しい働き方「LINE Hybrid Working Style」を採用しています。


■LINEに興味がある / マッチする職種があれば提案してほしい方はこちら