Data Lakehouse & Pipeline

A Data Lakehouse stores raw data and analytics-ready data in one system. Our Data Lakehouse & Pipeline brings data from your existing systems, files and data-collecting devices into this central repository. Data undergoes quality checks in layers, access can be controlled, and its origins can be traced. The same data can then serve reporting, analytics and AI, including Predictive Analytics and Enterprise KM Chatbot.

Many organisations store data in separate places: ERP/MES databases, Excel files, NAS devices and production-line equipment. Integration often relies on scheduled scripts that are hard to maintain, slowing reports and making figures difficult to trace. Starting an AI project then requires collecting and cleaning the data again each time.

• Bring data together — connect factory equipment, Inspektion apps sending data through nimitFlow, and existing ERP/MES to one central repository.
• Reduce Data Silos — turn data isolated across systems into a single, quality-checked dataset.
• Provide a foundation for other systems — supply Gold-layer data to Predictive Analytics, Enterprise KM Chatbot and BI tools.

Data Lakehouse & Pipeline works in 5 steps: connect sources, preserve raw data, validate and clean it, prepare it for use, and let users query it. Data-layer software is Open Source, while nimitFlow and Prometheus/Grafana monitor the system.

1. 1

   Connect data sources

   Ingest SQL databases (such as Oracle, PostgreSQL and MS SQL), APIs, files, Google Sheets and IoT devices using Apache Airflow for batch jobs and Apache Kafka for continuous streams or source-change capture (CDC).

2. 2

   Bronze — preserve raw data

   Preserve data in its source format (Schema-on-read), so it can be reprocessed when rules change.

3. 3

   Silver — validate and clean

   Apache Spark validates, cleans and deduplicates data according to Data Quality Rules, then stores it in Iceberg tables.

4. 4

   Gold — data ready for use

   Aggregate and enrich data for business needs, ready for BI reports, analytics and AI/ML.

5. 5

   Use data and monitor the system

   Users query data with SQL through Trino / Dremio and connect Tableau, Power BI or Jupyter Notebook. IT teams monitor all layers in one view using nimitFlow with Prometheus/Grafana and receive LINE or email alerts.

Storage is separate from Compute, so capacity and processing resources can scale independently with workload without stopping the system. Iceberg tables support ACID updates (each transaction is committed fully or not at all), Time Travel to earlier data states, and schema changes without rebuilding tables.

In this architecture, data resides on your organisation’s servers (On-Premise), and users can access only what their roles permit. Centralised data needs tighter access control, so authentication, authorisation, encryption and access logging are designed in from the start, as listed below.

• Authentication — Keycloak connects Active Directory / LDAP accounts and provides Single Sign-On for organisational apps through OIDC/SAML (shared-login standards), with Multi-Factor Authentication (MFA).
• Authorisation — Apache Ranger centrally sets database-, table-, column- and row-level permissions according to user roles (Role-Based Access Control, RBAC).
• Protect sensitive data — mask or replace personal data at column level (Masking / Tokenization).
• Encryption — stored data uses AES-256; data transfers use TLS 1.2 or later.
• Auditing — log every access and permission change, with forwarding to your organisation’s security monitoring system (SIEM).
• Data origins — OpenMetadata provides a Data Catalog, Business Glossary and column-level Data Lineage.

A Data Lakehouse suits organisations with data across many systems that need reports based on the same data and must keep it in-house. It may be unnecessary when data volumes or sources are limited; a central database or nimitFlow may be enough to start. If machines do not yet collect data, you first need collection devices or a source system, such as InSensor for sensor data. We assess data volumes and source-system counts before proposing an approach.

We have integrated production and management data for an automotive parts manufacturer in Thailand and prepared transaction data to train Machine Learning models for a Thai financial-sector organisation. No project using the Data Lakehouse architecture shown here has been delivered yet; it comes from our latest proposal. The work below is therefore related data experience, not delivered Data Lakehouse projects.

• In 2022–2023, we moved production-data websites for 11 production lines to a factory’s central VMware server for an automotive parts manufacturer in Thailand. The factory’s IT team had separated the office and production networks under its security policy. Server work was carried out jointly with the customer’s IT team, which defined IP addresses, ports and installation permissions.
• In 2023–2024, we developed a central-database web app on Google Cloud for the same automotive parts manufacturer. It brought KPI, internal loss, scrap, output, sales and profit from multiple report files into one website. The first phase was formally accepted in 2023; the customer-claims module was still being refined and its data validated in early 2024.
• In 2021–2023, we developed a Machine Learning system for a Thai financial-sector organisation. Payment transaction data was cleaned and used to train models to predict response-time anomalies 5 and 10 minutes ahead. The system ran on the customer’s servers, separate from transaction processing. Project closure documents were prepared in 2022, followed by contractual maintenance.

What we learned

IT security policies determine where data resides and which networks can access it. In both data-integration projects for the automotive parts manufacturer, the customer’s IT team defined network settings, installation permissions and cloud-system security checks. Design therefore needs IT involvement from the start.

Figures imported from existing reports must be checked against the originals before users rely on them for decisions. Data validation and user-feedback changes continue after the first delivery and need to be planned from the outset. AI work starts with cleaning data before model training, and new systems should be separated from core operations to avoid service disruption.

What does it mean to you?

Your Data Lakehouse project starts by reviewing source systems and security policies with your IT team. We then build a Bronze-to-Gold pipeline using a pilot dataset. You compare Gold-layer figures with your existing reports, and only expand to other sources once the figures match.

Project stages and deliverables

We plan Data Lakehouse projects in 5 stages: detailed design; server and platform installation; data integration and security configuration; testing and go-live; and post-launch support with knowledge transfer. Intensive post-launch support (Hypercare) and knowledge transfer can be included in the agreed scope so your IT team can take over operation.

1. 1

   Detailed design

   Confirm requirements; prepare high- and low-level designs (HLD/LLD), a permissions matrix, data-retention policies and a test plan.

2. 2

   Install servers and the platform

   Install hardware and Linux, harden security against the CIS Benchmark, then install platform components.

3. 3

   Integrate data and configure security

   Build a Bronze-to-Gold pipeline with a pilot dataset, connect SSO, and configure permissions, encryption and auditing.

4. 4

   Test and go live

   Run user acceptance testing (UAT), resolve defects, then deploy to production.

5. 5

   Post-launch support and knowledge transfer

   Define the post-launch support period and workshops for each team in the agreed scope.

Handover documents are in Thai, retaining English technical terms: system designs, installation guides, administration manuals, daily-operation guides, monitoring manuals, alert-configuration documents, troubleshooting guides, UAT records, As-Built documentation and training materials.