Lead Data Engineer (InSights)
OPSWAT
Protecting the World’s Critical Infrastructure
OPSWAT, a global leader in IT, OT, and ICS critical infrastructure cybersecurity, delivers an end-to-end platform that gives public and private sector organizations and enterprises the critical advantage needed to protect their complex networks, secure their devices, and ensure compliance. Over the last 20 years our commitment to innovative technology has earned the trust of more than 1,700 organizations, governments, and institutions globally, solidifying our role in protecting the world’s critical infrastructure and securing our way of life.
The Position
OPSWAT is looking for a Lead Data Engineer to take charge of our data engineering function for our MetaDefender Insights Threat Intelligence Offerings. This role will be focused on shaping the technical implementations and execution for a growing data engineering team. This role will be pivotal not only in building and managing scalable data pipelines but also in setting the technical direction for data architecture, processes, and feature sets. The ideal candidate will have experience with modern data platforms like Prefect, Airflow, and others, as well as expertise in streaming analytics, data enrichments, and a variety of data structures and data stores that will enable a fluid and dynamic discussion with stakeholders and Technical Product Management. You will be responsible for both hands-on technical work and aiding in the development of a technical roadmap to grow the team as our data capabilities and focus expands.
What You Will Be Doing
- Design, build, and maintain robust data pipelines for real-time and batch processing using platforms like Prefect, Airflow, etc.
- Develop and optimize data architectures that support data ingestion, transformation, enrichment, and storage, ensuring scalability and efficiency.
- Lead efforts in streaming analytics, enabling real-time insights and predictive modeling across diverse data sets with a focus on Infrastructure Analysis (internet scale) and threat actor tracking.
- Help define the technical vision for the data engineering function, including strategy for team expansion, tools selection, and best practices.
- Collaborate with product management to ensure data engineering solutions and approaches align with overall business goals and analytics needs both short and long term.
- Mentor and guide junior data engineers, establishing a culture of collaboration, continuous learning, and innovation.
- Implement monitoring, logging, and alerting for data pipelines to ensure reliability, quality, and accuracy of data outputs.
What We Need from You
- 5+ years of experience in data engineering with hands-on expertise in tools like Prefect, Airflow, and similar platforms.
- Proven experience with streaming analytics, data transformations, data structures, and various data storage solutions (SQL, NoSQL, etc.).
- Strong proficiency in Python, GO, Rust, SQL, and experience with cloud-based data ecosystems (AWS, Azure, GCP, etc.).
- Experience in leading or mentoring data engineering teams, with a demonstrated ability to define technical roadmaps and guide team growth.
- Excellent written and verbal communication skills in English.
- Ability to collect, organize, analyze, and present complex data with high attention to detail and accuracy.
- Ability to present technical topics to non-technical audiences, as you may be asked to present at conferences and to customers on our technical approaches and analytic processes.
It Would Be Nice if You Had
- Experience with Threat Intelligence and Cyber security problem domains.
- Experience with additional data orchestration tools and frameworks.
- Knowledge of Kafka, Kinesis, or other streaming platforms.
- Advanced capabilities in data visualization tools like PowerBI, Tableau, or similar.
- Familiarity with infrastructure-as-code tools like Terraform or CloudFormation for deploying and managing data infrastructure.
- A passion for driving data-driven decisions and fostering a collaborative data culture.
OPSWAT is an equal opportunity employer. We celebrate diversity and are committed to providing an environment where equal employment opportunities are extended to all employees and applicants, free of discrimination and harassment of any type. All employment decisions are based on individual qualifications, job requirements, and business needs without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other category protected by federal, state, or local laws.
Recruiting Agencies: we do not accept unsolicited resumes from third party agencies for any of our open positions. To submit resumes for our jobs, there must be a recruiting contract approved by our legal team and endorsed by both parties. We are currently not accepting additional 3rd party agencies at this time.