Renato is available for hire

Renato Pedroso Neto

Verified Expert in Engineering

Data Engineer and Developer

Location

São Paulo - State of São Paulo, Brazil

Toptal Member Since

April 14, 2022

Renato拥有超过13年的大数据项目经验. 他曾任职于Databricks、Capco和金融机构. Renato已经将数拍字节的数据迁移到本地和云数据湖环境, architected entire lakehouses, 实施机器学习模型，为客户提供智能建议，管理多元文化数据团队，为巴西一流银行提供数据项目. He has a master's degree in big data.

Data Analysis Data Analytics Big Data Data Engineering Machine Learning Data Warehousing Databases Spark Python SQL PySpark ETL Apache Spark Amazon Web Services (AWS)Databricks Databricks Platform Logic Apache Superset

Portfolio

Databricks

Spark, Databricks，大数据，客户关系管理，Redash, Delta Lake...

Comniscient Technologies LLC dba Comlinkdata

数据工程，数据管道，Python, Scala, Spark, Amazon Athena...

An Online Freelance Agency

Spark, Python, Apache Kafka, Cloud, Amazon Web Services (AWS)， PySpark, ETL...

Experience

Python - 8 years Big Data - 8 years SQL - 8 years Spark - 8 years Data Engineering - 8 years Data Lakes - 8 years Data Science - 6 years Databricks - 1 year

Availability

Part-time

Preferred Environment

Spark, Databricks, Python, Amazon Web Services (AWS), Google Cloud Platform (GCP), Machine Learning, Big Data, Amazon Elastic MapReduce (EMR), SQL, Amazon RDS

The most amazing...

...这个项目是为巴西的一个开放银行数据采集项目，该项目使用机器学习来保证质量，并为金融机构提供一个良好的数据源.

Work Experience

Delivery Solutions Architect

2021 - PRESENT

Databricks

通过分析客户数据并提出改进建议，将客户使用率提高了2倍.
对Spark环境进行压力测试，在28分钟内生成和散列1万亿行.
获得AWS解决方案架构师和Spark开发人员认证.

Technologies: Spark, Databricks，大数据，客户关系管理，Redash, Delta Lake, Python, Amazon Web Services (AWS), PySpark, ETL, Data Lakes, Apache Spark, Data, Data Analysis, Data Analytics, Business Intelligence (BI), Amazon S3 (AWS S3), Amazon EC2, Snowflake, AWS Glue, ELT, Databases

Data Engineer

2022 - 2023

Comniscient Technologies LLC dba Comlinkdata

开发电信市场数据和洞察平台的新指标, 使用Spark来帮助客户理解客户的行为.
帮助构建和发展一个产品，以检查网络运营商在一个国家的竞争力.
在Airflow中实现了使用Spark转换电信数据的新dag.

Technologies: 数据工程，数据管道，Python, Scala, Spark, Amazon Athena, Amazon Web Services (AWS), Data Analytics, Business Intelligence (BI), Redshift, Data Warehousing, Amazon S3 (AWS S3), Amazon EC2, AWS Glue, ELT, Databases

Data Engineer

2022 - 2022

An Online Freelance Agency

Worked with a client to architect, construct, 并支持从内部部署到云环境的数据管道.
Rearchitected the client's data pipeline in the cloud, reducing the total cost of ownership (TCO) by 40%.
提供Python代码方面的咨询，包括一般指导和最佳实践.

Technologies: Spark, Python, Apache Kafka, Cloud, Amazon Web Services (AWS)， PySpark, ETL, Data Lakes, Apache Spark, Data, Data Analysis, Data Analytics, Business Intelligence (BI), Redshift, Data Warehousing, Amazon S3 (AWS S3), Amazon EC2, PostgreSQL, ELT, Databases

Lead Data Engineer | Architect | Scientist

2016 - 2021

Capco

标准化数据实践，并将其作为Capco官方产品发布.
拥有Capco咨询和创新实验室的所有数据项目.
领导开放式银行数据采集和标准化的开发，直接交付给金融机构.
为金融机构创建并调整了一个自然语言模型.
为Capco的客户开发市场数据管道.

Technologies: Google Cloud Platform (GCP), Python, Machine Learning, Data Engineering, Data Architecture, Big Data, Prototyping, Spark, People Management, Amazon Web Services (AWS), PySpark, ETL, Redshift, Data Lakes, Message Queues, Stream Processing, Apache Spark, Data, Data Analysis, Data Analytics, Business Intelligence (BI), Amazon S3 (AWS S3), Amazon EC2, PostgreSQL, MongoDB, AWS Glue, Predictive Modeling, ELT, Redis, Databases

Big Data Systems Engineer

2014 - 2016

Banco Itaú

将10PB的数据从大型机迁移到Hadoop环境，创建可靠的数据管道.
Delivered 99.HDFS环境下99%的数据可用性.
为整个银行建立了一个信息中心.
使用Spark的制度化并行处理，为业务领域提供快速的结果.

Technologies: Spark, Hadoop, Apache Hive, MySQL, Mainframe, PySpark, ETL, Data Lakes, Data Warehousing, Apache Spark, Data, Data Analysis, Data Analytics, Business Intelligence (BI), ELT, Databases

Experience

Open Banking Data Ingestion

开放银行数据摄取、清理和标准化. 该项目旨在获取巴西所有开放的银行数据，并向金融机构出售访问这些数据的订阅服务. 整个项目采用无服务器架构和并行处理原则在GCP中开发. 使用机器学习模型保证数据质量.

Financial Data Web Scraping

构建网络抓取软件，从巴西经纪人捕获数据. 这些数据被输入到MySQL数据库中，以便进一步分析，以帮助进行投资回测. 该项目使用本地Python对象计算线程和并行处理技术.

Beacon Data Analysis

信标(物联网)数据分析，预测atm机的客户行为和盗窃行为. 该原型分析了来自多个物联网设备的NRT数据，以跟踪银行分行内部的路径, survey ATMs, and inform the security team about abnormalities

Monolith Decomposition

Cobol代码分析和机器学习模型实现，以帮助金融机构分析将Cobol单体应用程序分解为有意义的微服务以实现平台现代化的最佳方法.

Sentiment Analysis for Financial Institutions

情感分析模型为金融机构服务. 这个想法是为银行训练一个模型，以便更好地从文本和语音中提取情感，以提高客户保留率和满意度.

Mainframe to Big Data Environment Engineering

由Cloudera提供的从大型机环境到内部部署Hadoop的庞大数据传输管道. 该项目包括创建数据质量层、摄取和向业务领域交付.

Education

2015 - 2016

Specialization in Data Science

约翰霍普金斯大学|通过Coursera -巴西圣保罗

2013 - 2015

Master's Degree in Big Data

巴西圣保罗Informática行政学院(FIAP)

2007 - 2011

Bachelor's Degree in Computer Science

Mackenzie University - Sao Paulo, Brazil

Certifications

JANUARY 2023 - PRESENT

Databricks Certified Machine Learning Professional

Databricks

MAY 2022 - PRESENT

Databricks Certified Data Engineer Professional

Databricks

FEBRUARY 2022 - FEBRUARY 2024

Databricks认证的Apache Spark 3助理开发人员.0

Databricks

JANUARY 2022 - JANUARY 2025

AWS Certified Solutions Architect Associate

AWS

FEBRUARY 2018 - PRESENT

Machine Learning Engineer

Udacity

OCTOBER 2016 - PRESENT

Data Science Specialization

Coursera

MAY 2016 - PRESENT

Getting and Cleaning Data

Coursera

MARCH 2016 - PRESENT

Dell EMC Data Science Associate (EMCDSA)

Dell EMC

MARCH 2014 - PRESENT

Linux Professional Institute 101 (LPIC-1)

Linux Professional Institute

Skills

Libraries/APIs

Spark Streaming, PySpark, Pandas, Scikit-learn, NumPy, Beautiful Soup, Selenium WebDriver

Tools

Git, Apache Airflow, Amazon Elastic MapReduce (EMR), Redash, BigQuery, Amazon Simple Queue Service (SQS), Amazon Transcribe, Amazon QuickSight, Amazon Athena, AWS Glue, Apache Maven

Frameworks

Spark, Apache Spark, Hadoop, Flask, Selenium, Scrapy

Paradigms

数据科学，ETL，商业智能(BI)，逻辑编程

Languages

Python, SQL, COBOL, XPath, Scala, Snowflake

Platforms

Databricks, Amazon Web Services (AWS)， Linux, Amazon EC2, Google Cloud Platform (GCP)， Apache Kafka

Storage

Databases, Apache Hive, Data Pipelines, Redshift, Data Lakes, Amazon S3 (AWS S3), NoSQL, MySQL, Google Cloud Datastore, PostgreSQL, MongoDB, Redis

Other

Machine Learning, Big Data, Data Engineering, Data Warehousing, Data, Data Analysis, Data Analytics, ELT, Systems Analysis, Cloud, Stream Processing, Scraping, Data Scraping, Web Scraping, Predictive Modeling, Amazon RDS, Operating Systems, IT Systems Architecture, Neural Networks, Statistics, Deep Learning, Data Modeling, Mainframe, Data Architecture, Prototyping, People Management, Client Relationship Management, Delta Lake, Google Cloud Functions, Pub/Sub, Vertex, Apache Superset, Clustering, Reporting, Natural Language Processing (NLP), APIs, Message Queues, GPT, Generative Pre-trained Transformers (GPT), Processing & Threading

Collaboration That Works

How to Work with Toptal

在数小时内，而不是数周或数月，我们的网络将为您直接匹配全球行业专家.

Share your needs

在与Toptal领域专家的电话中讨论您的需求并细化您的范围.

Choose your talent

在24小时内获得专业匹配人才的简短列表，以进行审查，面试和选择.

Start your risk-free talent trial

与你选择的人才一起工作，试用最多两周. Pay only if you decide to hire them.

Top talent is in high demand.

Start hiring