Article 5 Tokenization - Converting Text to Number

Article 5: Tokenization - Converting Text to Numbers for Neural Networks

Introduction: Why Tokenization Matters

Imagine trying to teach a computer to understand Shakespeare without first teaching it to read. This is the fundamental challenge of natural language processing. Computers speak mathematics, while humans speak words. Tokenization is the crucial bridge between these two worlds.

Every time you ask ChatGPT a question, search for information online, or get an auto-complete suggestion in your email, tokenization works silently behind the scenes. It converts your text into the numerical sequences that power these intelligent systems.

Continue reading

Article 11 - Dataset Curation and Training Languag

Building Custom Language Models: From Raw Data to AI Solutions

In today’s AI-driven world, the ability to create custom language models tailored to specific domains and tasks represents a critical competitive advantage. This comprehensive guide walks you through the complete lifecycle of building language models from the ground up—from curating high-quality datasets to training and refining powerful AI systems.

Whether you’re developing specialized models for healthcare, finance, legal services, or any domain requiring nuanced understanding, this chapter provides the practical knowledge and code examples you need to succeed. We’ll explore modern techniques using the Hugging Face ecosystem that balance efficiency, scalability, and model quality.

Continue reading

Streamlit Build Interactive Data Apps

Stop Wrestling with Static Reports: Build Interactive Data Apps with Streamlit

Ever felt that gut punch when your carefully crafted report lands with a thud? You have crunched the numbers, built charts, and sent a shiny PDF, only to be hit with: “Can you filter by region?” “What about Q2?” “Can I see product details?” Each question sends you back to your code, tweaking scripts, exporting files, and emailing report_final_v5.pdf. It is like mailing postcards when your team craves a live Zoom call.

Continue reading

Making Sense of Textract Output A Developer's Fast

Making Sense of Textract Output: A Developer’s Fast Track with the TRP Library

You know that feeling when you open a scanned document, and it’s like all the valuable information is just sitting there—but shattered across the page in a hundred disjointed fragments? Sure, traditional OCR gets you the text. But it doesn’t give you the map. It doesn’t tell you how the pieces fit together: what’s a table, what’s a form, what field goes with what value.

Continue reading

Amazon Textract A Developer's Guide

Unlock the hidden potential of your documents! Dive into our latest guide on Amazon Textract and discover how to transform unstructured data into actionable insights. From invoices to contracts, learn the secrets of document intelligence that could revolutionize your workflow. Don’t let your data stay trapped—read on to unleash its power!

Amazon Textract converts documents into structured data by detecting forms, tables, and layouts while enabling natural language queries. It includes expense and ID analysis APIs and handles both real-time and batch processing.

Continue reading

Unlocking Document Intelligence with Amazon Textract

Unlock the hidden potential of your documents. Discover how Amazon Textract is changing document processing by not just reading text but also understanding its structure and context.

Amazon Textract improves document processing by detecting structure and context. It automates the extraction of key data, tables, and query responses more efficiently than standard OCR.

Unlocking Document Intelligence with Amazon Textract and TRP Lib

Have you ever looked at a scanned document and felt that unique frustration? All the information is right there on your screen, but it is trapped in a format that is hard to extract. Traditional OCR (Optical Character Recognition) can read the words, but it does not understand how they fit together or what they mean in context.

Continue reading

Building Your First Intelligent Document Workflow

Tired of drowning in a sea of paperwork? Discover how to transform that mountain of PDFs into actionable insights with AWS’s intelligent document workflows! Say goodbye to chaos and hello to efficiency—your digital assistant awaits!

Learn to build an intelligent document workflow using AWS Textract and Amazon Comprehend to automate document processing, extract text, analyze content, and gain insights from unstructured data, transforming chaos into structured information.

Let’s explore how to use Textract’s FeatureTypes parameter to extract form data more precisely. The FORMS feature detects key-value pairs, while TABLES finds structured tabular data. And explore Comprehend’s ability to analyze sentiment and emotional tone in documents.

Continue reading

Unlocking Document Intelligence: How AWS is Transforming Document Processing

The Hidden Cost of Manual Document Processing

Picture this: A healthcare administrator manually entering patient intake forms. A financial analyst carefully extracting data from hundreds of invoices. A legal team searching through mountains of contracts to find specific clauses.

Does this sound familiar?

Despite our efforts to go digital, businesses in all industries still waste countless hours on manual document processing. According to industry studies, employees spend up to 30% of their time on document-related tasks. This is time that could be spent on more valuable work.

Continue reading

The Ultimate Guide to Text Embedding Models in 202

Looking to enhance your AI search capabilities? In 2025, embedding model selection is key for RAG systems and semantic search. This guide compares OpenAI, AWS, and open-source options to help you build more accurate, context-aware applications.

Text embedding models convert language into numerical representations, enabling powerful semantic search, recommendations, and RAG capabilities. Here’s how to choose the right model for your needs.

Choosing the right text embedding model is vital for NLP systems in 2025. Performance on specific tasks, technical specs, cost, and licensing are key factors to consider. While MTEB provides overall benchmarks, task-specific performance matters most for retrieval and RAG systems. OpenAI, AWS, and open-source options each offer distinct trade-offs.

Continue reading

Improving Search and RAG with Vectors and BM25

Combining Traditional Keyword Logic with Cutting-Edge Vector Text Embedding Technology

Why hybrid search—combining traditional keyword logic with cutting-edge vector technology—has become essential for any data-driven product.

You know the moment: you punch “latest Volvo electric SUV safety reviews” into a site search, and—despite the fact that you know the documents exist—you’re staring at page three of irrelevant hits. Classic keyword search has failed you, yet pure “AI” search often misses the exact phrase you needed. The fix isn’t more synonyms or a bigger model. It’s teaching your stack to think in both words and meaning at the same time.

Continue reading

Stop the Hallucinations Hybrid Retrieval with BM25

Tired of LLMs hallucinating instead of citing the exact information you need? Discover the secret sauce that combines traditional keyword search with cutting-edge vector retrieval, then tops it all off with two levels of rerank. Unlock the power of hybrid retrieval and transform your RAG systems. Don’t let your search stack be the weak link—read on to level up your game!

Stop the Hallucinations: Hybrid Retrieval Using BM25, pgvector, Embedding Rerank, LLM Rerank, and HyDE

Continue reading

The Evolving Data Landscape and Architectural Impe

The Evolving Data Landscape and Architectural Imperatives

Just as a 1920s city planner could not anticipate self-driving cars, today’s technical leaders face the challenge of designing data architectures for an uncertain future. Traditional data warehouses struggle to keep pace with exploding data sources and growing AI demands, forcing us to fundamentally rethink our approach to data management. This article explores not just what modern data architecture is, but why it’s crucial for business success in today’s rapidly evolving landscape.

Continue reading

Data Governance Turning Information into Business

In today’s data-driven world, effective data governance isn’t just a technical necessity—it’s a business advantage.Organizations that treat data as a strategic asset rather than just an IT concern are seeing measurable returns on their investment. This article explores how robust data governance drives profitability, reduces risk, and enhances business agility through practical frameworks and real-world examples.

Why Data Governance Matters to Your Bottom Line

Unlike finite resources, data grows in value when properly managed. Modern data governance provides your information is accurate, consistent, secure, and available for real-time decision making. This foundation enables:

Continue reading

Adopting GenAI for the Busy Executive

Slash Costs and Boost Loyalty with AI-Powered Documentation

Remember the early internet, when websites were mostly static “brochureware”? This evolved into e-commerce. The brochureware approach proved surprisingly effective for customer support. It allowed companies to put product documentation, HR manuals, and engineering notes online where people could reference them. Later, search capabilities were added, making this content more accessible. A fundamental challenge remained: search alone couldn’t bridge the gap between complex documentation and user needs.

Continue reading

Article Streamlit Part 3 - Form Validation Part 1

Article: Streamlit Part 3

Form Validation Part 1

A Roundhouse Kick into Streamlit Form Validation

Amid the rhythmic thuds of gloves hitting pads, Rick and Chris were immersed in their kickboxing class. Between combos, they exchanged thoughts—not just on perfecting their strikes but also on coding challenges. As they caught their breath, the conversation shifted to Streamlit and the importance of form validation.

Rick: Panting “You know, Chris, it’s like the saying ‘garbage in, garbage out.’ If I don’t validate the data properly in my Streamlit app, I can’t expect good results. I need to guard the gate and make sure only clean data gets through.”

Continue reading

The Kafka Ecosystem

This article appeared on LinkedIn on Feb 24th, 2018.

The Kafka Ecosystem - Kafka Core, Kafka Streams, Kafka Connect, Kafka REST Proxy, and the Schema Registry

Rick HightowerEngineering Consultant focused on AI

February 24, 2018

The core of Kafka is the brokers, topics, logs, partitions, and cluster. The core also consists of related tools like MirrorMaker. The aforementioned is Kafka as it exists in Apache.

The Kafka ecosystem consists of Kafka Core, Kafka Streams, Kafka Connect, Kafka REST Proxy, and the Schema Registry. Most of the additional pieces of the Kafka ecosystem comes from Confluent and is not part of Apache.

Continue reading

Is JParse Fast

This article originally appeared on LinkedIn on Feb 19th, 2024 by Rick Hightower

JParse: The most efficient JSON parser for the JVM yet!

Rick Hightower Engineering Consultant focused on AI

February 19, 2023

JParse

JParse, is the most efficient JSON parser for the JVM yet.

Why JParse?

JParse is the most efficient JSON parser for the JVM yet - it uses an index overlay to deliver lightning-fast parsing speeds.

Continue reading

Using ChatGPT, Embeddings, and HyDE to Improve Search Results

This article originally appeared on LinkedIn on July 11th, 2023.

Using ChatGPT, Embeddings, and HyDE to Improve Search Results

Rick Hightower Engineering Consultant focused on AI

July 11, 2023

Using ChatGPT, Embeddings, and HyDE to Improve Search Results

Introduction

In today’s fast-paced business world, it is essential to stay ahead of the competition. An efficient search engine that can provide accurate information to your customers or employees can make a big difference. However, building and maintaining a robust search engine can be a challenge. In this dev notebook, we will explore how ChatGPT, Embeddings, and HyDE can help you improve your search results.

Continue reading

Advanced SQL Techniques for ETL

mindmap
  root((Advanced SQL Techniques for ETL))
    CASE Statements
      Conditional Logic
      Data Standardization
      Conditional Aggregation
      FILTER Alternative
    GROUP BY Operations
      Data Aggregation
      Monthly Metrics
      Handling Nulls
      COALESCE Function
    Window Functions
      Partitioning
      RANK
      Rolling Aggregates
      Duplicate Detection
    SQL Functions
      LEAD & LAG
      ROW_NUMBER
      DENSE_RANK
      Cumulative Metrics
    Table Partitioning
      Performance Optimization
      Date-Based Partitioning
      Query Efficiency
      Scalability

Advanced SQL Techniques for ETL

• CASE Statements with conditional logic, standardization, and FILTER alternatives
• GROUP BY Operations including aggregation, metrics, and null handling
• Window Functions with partitioning, ranking, and duplicate detection
• SQL Functions like LEAD(), LAG(), and ROW_NUMBER()
• Table Partitioning for performance and scalability

Ever wrestled with massive datasets using procedural scripts? You know that feeling—like moving a mountain with a teaspoon. Transform that struggle into power with advanced SQL techniques that turn hours into minutes.

Continue reading