¶ Data Product Creation Workflow
¶ Prerequisites
- User account with CREATE_DATA_PRODUCTS permission
- Assigned to a domain
- Understanding of data source
¶ Step-by-Step Creation Process
¶ Step 1: Meta (Basic Information)
-
Navigate to Data Products
- Click "Data Products" → "Create New"
-
Fill Basic Information:
Name: Customer Transaction History Key: customer-transaction-history (auto-generated) Description: Complete history of customer transactions including purchases, returns, and refunds Domain: Finance Classification: INTERNAL Update Frequency: Daily -
Select Transformer:
- None: Raw data, no transformation
- SQL: SQL-based transformations
-
Add Owner & Stewards:
Data Owners: [finance-team@company.com] Data Stewards: [data-engineering@company.com] -
Click "Next" to proceed to schema definition
¶ Step 2: Schema Method
Choose how to define your product schema:
Option 1: Manual Creation
- Type schema manually
- Best for simple schemas
- Full control over structure
Option 2: JSON/CSV Upload
- Upload sample file
- Schema extracted automatically
- Quick for existing files
Option 3: Multi-Datasource
- Connect to multiple databases
- PostgreSQL, MySQL, Oracle, SQL Server
- Visual ER Diagram Builder
- Join tables across databases
Option 4: Staging Connector
- Use Elementrix staging area
- For unsupported sources
- Custom ingestion logic
Option 5: S3 Connector
- Connect to S3 bucket
- Import from CSV, JSON, Parquet, Avro
- Schema extracted automatically
- Supports incremental updates
Select your method and click "Next"
¶ Step 3: Manual / Upload / Multi-Datasource / Staging Connector / S3 Connector
(Depends on chosen method)
If Manual:
-
Click "Add Field"
-
Define each field:
Field Name: transaction_id Data Type: UUID Required: Yes Description: Unique transaction identifier
If Upload:
- Upload CSV or JSON file
- Review detected schema
- Adjust field types if needed
- Confirm schema
If Multi-Datasource:
-
Add Datasource:
Name: [Customer Database] Type: [PostgreSQL / MySQL / Oracle / SQL Server] Host: [db.company.com] Port: [5432] Database: [customers_db] Username: [readonly_user] Password: [••••••••••] SSL Mode: [Require] -
Test Connection
-
Save Datasource
-
Add additional datasources as needed
-
Select tables from each datasource
-
Configure join relationships using ER Diagram Builder:
- Select join type (INNER, LEFT, RIGHT, FULL) - Map source and target columns - Set relationship cardinality -
Preview the combined schema
If Staging Connector:
- Define staging area path
- Specify file format
- Define ingestion schedule
- Configure data validation
If S3 Connector:
-
Configure AWS credentials:
Access Key ID: [AKIAIOSFODNN7EXAMPLE] Secret Access Key: [•••••••••••••••••••••] Region: [us-east-1] Authentication Method: - Access Keys - IAM Role (for EC2/ECS) - Assume Role -
Select bucket: [elementrix-data-lake]
-
Browse to path: [data/products/]
-
Select file format:
- CSV (auto-detect schema) - JSON (auto-detect schema) - Parquet (built-in schema) - Avro (built-in schema) -
Review detected schema
-
Configure incremental load settings:
Mode: Full refresh / Incremental / Merge Detect new files by: Modification time / Filename pattern
¶ Step 4: Fields Schema
Configure individual fields and quality attributes:
For Each Field:
-
Basic Properties:
Field Name: transaction_amount Display Name: Transaction Amount Data Type: Decimal(10,2) Required: Yes -
Quality Attributes:
☑ Unique: No ☑ Nullable: No ☑ PII: No ☑ Sensitive: No Validation Rules: - Minimum: 0.00 - Maximum: 1000000.00 - Format: ##,###.00 -
Business Context:
Description: Total transaction value in USD Business Rules: Includes taxes and fees Default Value: 0.00 Example Values: 49.99, 1299.00, 15.50 -
Data Quality Checks:
☑ Check for null values ☑ Validate range (0 - 1M) ☑ Format validation ☐ Custom validation rule
Repeat for all fields, then click "Next"
¶ Step 5: Workflow
Configure approval workflow for access requests:
Default Workflow:
Request → Owner Approval → Access Granted
Configure Approvers:
Step 1: Data Steward Review
Approvers: [data-steward@company.com]
Step 2: Owner Approval
Approvers: [owner@company.com]
Step 3: Governance Review (if sensitive)
Approvers: [governance@company.com]
Click "Next" when configured
¶ Step 6: Semantic Registry
Link business glossary terms to fields:
-
Search for Terms:
Search: "transaction" Results: - Transaction ID - Transaction Amount - Transaction Type - Transaction Date -
Map Terms to Fields:
Field: transaction_id Linked Term: Transaction ID Field: transaction_amount Linked Term: Transaction Amount Field: transaction_date Linked Term: Transaction Timestamp -
Add Product-Level Terms:
Data Product-Level Terms: - Customer Transaction - Financial Data - Transaction History -
Review Mappings:
- Ensure all important fields mapped
- Verify term definitions match usage
- Add new terms if needed
Click "Next" to continue
¶ Step 7: Edit Public Page
Create marketing content for data product discovery:
Marketing Page Sections:
-
Overview:
# Customer Transaction History Complete transactional data for all customer purchases, returns, and refunds. **Use Cases**: - Revenue analysis - Customer behavior insights - Fraud detection - Chargeback management -
Key Features:
✓ Real-time transaction data ✓ 7-year historical archive ✓ 99.9% uptime SLA ✓ Sub-second query response ✓ Daily data refresh -
Sample Queries:
-- Get transactions for customer SELECT * FROM transactions WHERE customer_id = '123e4567-e89b-12d3-a456-426614174000' ORDER BY transaction_date DESC; -- Calculate daily revenue SELECT DATE(transaction_date) as date, SUM(transaction_amount) as total_revenue FROM transactions WHERE transaction_date >= CURRENT_DATE - 30 GROUP BY DATE(transaction_date); -
Data Quality Metrics:
Completeness: 99.8% Accuracy: 99.95% Timeliness: < 5 minutes lag Consistency: 100% -
Support Information:
Contact: data-support@company.com Documentation: https://docs.company.com/transactions Slack Channel: #data-transactions
Click "Next" when complete
Result:
✓ Data Product Created Successfully!
Status: DRAFT
Next Steps:
1. Review product details
2. Test API endpoint
3. Submit for review when ready
¶ Next Steps
- Submit for Governance Approval
- Understand the Data Product Lifecycle
- Learn about Sync Strategies for different data sources