NewsHub - AI-Powered News Aggregation Platform

 


What I Built

NewsHub is an intelligent news aggregation platform that revolutionizes how users discover and consume news content. Built with Redis 8 as the core real-time data layer, NewsHub combines the power of AI with Redis's advanced capabilities to deliver personalized, contextually relevant news experiences.

This was my first time working with Redis, and the learning curve was steep but incredibly rewarding. I discovered that Redis is far more than just a cache - it's a powerful, multi-model platform capable of handling complex AI workloads. Building NewsHub taught me how Redis can serve as a primary database, vector search engine, and real-time analytics platform all in one.

Key Features

AI-Powered Content Intelligence

  • Smart summarization using Google Gemini AI for concise, meaningful article summaries
  • Advanced sentiment analysis with real-time emotion detection (positive, negative, neutral)
  • Dynamic keyword extraction and AI-driven topic identification
  • Semantic understanding for context-aware content analysis

Advanced Search & Discovery

  • Vector semantic search using Redis-based similarity matching with 768-dimensional embeddings
  • Multi-faceted filtering by topic, sentiment, source, keywords, and date ranges
  • Similar articles engine using cosine similarity algorithms
  • Fuzzy search with typo tolerance and intelligent suggestions

Intelligent Personalization

  • Smart feed management that automatically removes viewed articles from personalized feeds
  • Behavioral analytics tracking reading patterns and engagement metrics
  • Real-time recommendations based on user interaction patterns

Real-Time Analytics & Insights

  • Performance monitoring with cache hit rates, response times, and system health
  • Comprehensive user journey analytics

Performance & Scalability

  • Multi-layer caching with intelligent request, query, and result caching strategies
  • Sub-50ms response times through optimized Redis operations
  • Real-time processing with live news ingestion and AI analysis pipeline
  • Built to handle 1000+ concurrent users with horizontal scaling

Architecture Design

NewsHub follows a modern microservices architecture with Redis 8 serving as the central intelligence layer. The system is designed for high performance, scalability, and real-time AI processing.

High-Level System Architecture

System Aritecture

Redis 8 Multi-Model Platform Architecture

Redis Architecture

Data Flow & Processing Pipeline

Data Flow Diagram

Demo

Live Application

Youtube Demo


Audio & Accessibility Note:
Please forgive any background noise or my accent in the video. For the best experience, I recommend turning on closed captions. They're available in English and will help make sure you don't miss a thing!

Screenshots

Home Page - Intelligent News Feed

NewsHub Homepage

The main dashboard featuring AI-curated news with sentiment indicators, engagement metrics, and personalized recommendations.

Advanced Search Interface

Advanced Search

Powerful search capabilities with real-time filtering by topic, sentiment

Personalized Article View

Article Details

Detailed article view with AI-generated summaries, sentiment analysis, and similar article recommendations.

Personalized "For You" Feed

Personalized Feed

AI-curated personalized news feed that automatically adapts based on user preferences and reading behavior.

Similar Articles Discovery

Similar Articles

Vector-based semantic similarity engine showing contextually related articles using Redis vector search.

Key Features Demonstrated

  • Intelligent search with vector similarity search achieving sub-100ms response times
  • Real-time personalization with dynamic content curation based on user preferences
  • Smart article removal that automatically removes viewed articles from personalized feeds
  • Sentiment-based filtering with visual sentiment indicators
  • AI-powered trending detection with engagement metrics

How I Used Redis 8

Redis 8 serves as the primary real-time data layer powering NewsHub's intelligent features. Coming from a background where I primarily used traditional databases, learning Redis was eye-opening. I discovered how Redis can handle complex AI workloads while maintaining incredible performance.

Vector Search & Semantic Intelligence

This was the most challenging part of my Redis journey. Understanding how to implement vector search and semantic similarity was completely new to me, but the results were amazing.

// redisService.js - Vector Search Implementation
class RedisService {
  async createVectorIndex() {
    try {
      // Create vector search index for semantic similarity
      await this.client.ft.create('news:vector_index', {
        '$.title': 'TEXT',
        '$.content': 'TEXT',
        '$.embedding': 'VECTOR',
        '$.sentiment': 'TAG',
        '$.topic': 'TAG',
        '$.source': 'TAG',
        '$.publishedAt': 'NUMERIC'
      }, {
        ON: 'JSON',
        PREFIX: 'news:articles:',
        FILTER: '@publishedAt:[0 +inf]',
        SCHEMA: {
          '$.title': { type: 'TEXT', WEIGHT: 2.0 },
          '$.content': { type: 'TEXT', WEIGHT: 1.0 },
          '$.embedding': { 
            type: 'VECTOR', 
            DIM: 768, 
            DISTANCE_METRIC: 'COSINE',
            TYPE: 'FLOAT32'
          },
          '$.sentiment': { type: 'TAG' },
          '$.topic': { type: 'TAG' },
          '$.source': { type: 'TAG' },
          '$.publishedAt': { type: 'NUMERIC' }
        }
      });
      console.log('✅ Vector search index created successfully');
    } catch (error) {
      console.log('Vector index already exists or error:', error.message);
    }
  }

  async findSimilarArticles(articleId, limit = 10) {
    try {
      // Get article embedding
      const article = await this.client.json.get(`news:articles:${articleId}`);
      if (!article || !article.embedding) {
        throw new Error('Article or embedding not found');
      }

      // Vector similarity search
      const query = `*=>[KNN $k @embedding $embedding AS score]`;
      const params = {
        k: limit,
        embedding: article.embedding
      };

      const results = await this.client.ft.search('news:vector_index', query, {
        PARAMS: params,
        RETURN: ['$.title', '$.content', '$.sentiment', '$.topic', 'score'],
        SORTBY: 'score',
        DIALECT: 2
      });

      return results.documents.map(doc => ({
        id: doc.id.replace('news:articles:', ''),
        title: doc.value['$.title'],
        content: doc.value['$.content'],
        sentiment: doc.value['$.sentiment'],
        topic: doc.value['$.topic'],
        similarity: doc.value.score
      }));
    } catch (error) {
      console.error('Error finding similar articles:', error);
      throw error;
    }
  }
}

JSON Storage with Full-Text Search

Learning to use Redis as a primary database was a paradigm shift. The ability to store complex JSON documents while maintaining full-text search capabilities was incredible.

// redisService.js - JSON Storage Implementation
class RedisService {
  async storeArticle(article) {
    try {
      const articleId = `news:articles:${article.id}`;

      // Store article as JSON with full-text search capabilities
      await this.client.json.set(articleId, '$', {
        id: article.id,
        title: article.title,
        content: article.content,
        summary: article.summary,
        sentiment: article.sentiment,
        topic: article.topic,
        source: article.source,
        url: article.url,
        publishedAt: article.publishedAt,
        embedding: article.embedding,
        keywords: article.keywords,
        metadata: {
          wordCount: article.wordCount,
          readingTime: article.readingTime,
          language: article.language
        }
      });

      // Create search index for full-text search
      await this.createSearchIndex();

      console.log(`✅ Article stored: ${articleId}`);
      return article.id;
    } catch (error) {
      console.error('Error storing article:', error);
      throw error;
    }
  }

  async searchArticles(query, filters = {}) {
    try {
      let searchQuery = `(@title:${query} | @content:${query} | @summary:${query})`;

      // Add filters
      if (filters.sentiment) {
        searchQuery += ` @sentiment:{${filters.sentiment}}`;
      }
      if (filters.topic) {
        searchQuery += ` @topic:{${filters.topic}}`;
      }
      if (filters.source) {
        searchQuery += ` @source:{${filters.source}}`;
      }
      if (filters.dateRange) {
        searchQuery += ` @publishedAt:[${filters.dateRange.start} ${filters.dateRange.end}]`;
      }

      const results = await this.client.ft.search('news:search_index', searchQuery, {
        RETURN: ['$.title', '$.content', '$.sentiment', '$.topic', '$.source'],
        SORTBY: '@publishedAt',
        LIMIT: { from: 0, size: 50 }
      });

      return results.documents.map(doc => ({
        id: doc.id.replace('news:articles:', ''),
        title: doc.value['$.title'],
        content: doc.value['$.content'],
        sentiment: doc.value['$.sentiment'],
        topic: doc.value['$.topic'],
        source: doc.value['$.source']
      }));
    } catch (error) {
      console.error('Error searching articles:', error);
      throw error;
    }
  }
}

Multi-Layer Caching Strategy

Understanding Redis caching patterns was crucial for performance. I implemented a sophisticated multi-layer approach that dramatically improved response times.

// cacheClearService.js - Multi-layer Caching Implementation
class CacheClearService {
  constructor(redisClient) {
    this.client = redisClient;
    this.cacheLayers = {
      REQUEST: 'cache:requests:',
      QUERY: 'cache:queries:',
      RESULT: 'cache:results:',
      SIMILARITY: 'cache:similarity:',
      USER: 'cache:user:'
    };
  }

  async setCacheWithTTL(key, data, ttl = 3600) {
    try {
      const cacheKey = `cache:results:${key}`;
      await this.client.json.set(cacheKey, '$', {
        data: data,
        timestamp: Date.now(),
        ttl: ttl
      });
      await this.client.expire(cacheKey, ttl);
      console.log(`✅ Cache set: ${cacheKey}`);
    } catch (error) {
      console.error('Error setting cache:', error);
    }
  }

  async getCache(key) {
    try {
      const cacheKey = `cache:results:${key}`;
      const cached = await this.client.json.get(cacheKey);

      if (cached && cached.data) {
        console.log(`✅ Cache hit: ${cacheKey}`);
        return cached.data;
      }

      console.log(`❌ Cache miss: ${cacheKey}`);
      return null;
    } catch (error) {
      console.error('Error getting cache:', error);
      return null;
    }
  }
}

Real-Time Personalization Engine

The personalization system was where I learned about Redis's advanced data structures. Implementing automatic article removal from personalized feeds was a great learning experience.

// redisService.js - User Preferences Implementation
class RedisService {
  async storeUserPreferences(userId, preferences) {
    try {
      const userKey = `user:preferences:${userId}`;

      await this.client.hset(userKey, {
        topics: JSON.stringify(preferences.topics),
        sources: JSON.stringify(preferences.sources),
        sentiment: preferences.sentiment,
        language: preferences.language,
        updatedAt: Date.now()
      });

      // Set expiration for user preferences (30 days)
      await this.client.expire(userKey, 30 * 24 * 60 * 60);

      console.log(`✅ User preferences stored: ${userKey}`);
      return true;
    } catch (error) {
      console.error('Error storing user preferences:', error);
      throw error;
    }
  }

  async getPersonalizedNews(userId, limit = 20) {
    try {
      const preferences = await this.getUserPreferences(userId);
      if (!preferences) {
        return await this.getTrendingArticles(limit);
      }

      // Build personalized search query
      let searchQuery = '*';
      const filters = [];

      if (preferences.topics.length > 0) {
        filters.push(`@topic:{${preferences.topics.join('|')}}`);
      }
      if (preferences.sources.length > 0) {
        filters.push(`@source:{${preferences.sources.join('|')}}`);
      }
      if (preferences.sentiment) {
        filters.push(`@sentiment:{${preferences.sentiment}}`);
      }

      if (filters.length > 0) {
        searchQuery = filters.join(' ');
      }

      const results = await this.client.ft.search('news:search_index', searchQuery, {
        RETURN: ['$.title', '$.content', '$.sentiment', '$.topic', '$.source'],
        SORTBY: '@publishedAt',
        LIMIT: { from: 0, size: limit }
      });

      return results.documents.map(doc => ({
        id: doc.id.replace('news:articles:', ''),
        title: doc.value['$.title'],
        content: doc.value['$.content'],
        sentiment: doc.value['$.sentiment'],
        topic: doc.value['$.topic'],
        source: doc.value['$.source']
      }));
    } catch (error) {
      console.error('Error getting personalized news:', error);
      throw error;
    }
  }

  // Auto-remove viewed articles from personalized feed
  async trackArticleView(userId, articleId) {
    try {
      // Add to viewed articles set
      await this.client.sadd(`user:viewed:${userId}`, articleId);

      // Remove from personalized feed
      await this.client.srem(`user:personalized:${userId}`, articleId);

      // Track engagement metrics
      await this.trackEngagement(articleId, 'view');

      console.log(`✅ Article view tracked and removed from personalized feed: ${articleId}`);
    } catch (error) {
      console.error('Error tracking article view:', error);
    }
  }
}

Trending Articles & Analytics

Learning to use Redis Sorted Sets for trending algorithms was fascinating. The ability to implement time-decay algorithms directly in Redis was a game-changer.

// redisService.js - Trending Articles Implementation
class RedisService {
  async updateTrendingArticles() {
    try {
      // Get all articles with engagement metrics
      const articles = await this.client.ft.search('news:search_index', '*', {
        RETURN: ['$.title', '$.source', '$.publishedAt'],
        SORTBY: '@publishedAt',
        LIMIT: { from: 0, size: 1000 }
      });

      const trendingScores = {};

      for (const doc of articles.documents) {
        const articleId = doc.id.replace('news:articles:', '');
        const publishedAt = parseInt(doc.value['$.publishedAt']);
        const now = Date.now();

        // Calculate trending score based on recency and engagement
        const timeDecay = Math.exp(-(now - publishedAt) / (24 * 60 * 60 * 1000)); // 24 hours
        const engagementScore = await this.getEngagementScore(articleId);

        trendingScores[articleId] = timeDecay * engagementScore;
      }

      // Store trending articles in sorted set
      const trendingKey = 'trending:articles';
      await this.client.del(trendingKey); // Clear existing

      for (const [articleId, score] of Object.entries(trendingScores)) {
        await this.client.zadd(trendingKey, score, articleId);
      }

      console.log(`✅ Updated trending articles: ${Object.keys(trendingScores).length} articles`);
    } catch (error) {
      console.error('Error updating trending articles:', error);
      throw error;
    }
  }

  async trackEngagement(articleId, action = 'view') {
    try {
      const engagementKey = `metrics:engagement:${articleId}`;
      const actionKey = `${action}:count`;

      await this.client.hincrby(engagementKey, actionKey, 1);
      await this.client.hset(engagementKey, 'last_updated', Date.now());

      // Set expiration for engagement metrics (7 days)
      await this.client.expire(engagementKey, 7 * 24 * 60 * 60);

      console.log(`✅ Tracked engagement: ${action} for article ${articleId}`);
    } catch (error) {
      console.error('Error tracking engagement:', error);
    }
  }
}

Performance Results

Through my Redis learning journey, I achieved impressive performance metrics:

Cache Performance

  • Cache hit rate of 85-90% for frequently accessed data
  • Response times under 50ms for cached responses
  • Optimized memory usage with LRU eviction policies

Vector Search Performance

  • Similarity search under 100ms for 1000+ articles
  • 768-dimensional vector storage with COSINE distance
  • Efficient compression with RedisJSON

Scalability Metrics

  • Supports 1000+ concurrent users
  • Efficient JSON storage with full-text search capabilities
  • Sub-second response times for personalized feeds

Learning Experience

This project was my introduction to Redis, and it completely changed my perspective on data storage and caching. I learned that Redis isn't just a cache - it's a complete data platform capable of:

  • Serving as a primary database with JSON documents
  • Powering vector search for AI applications
  • Handling real-time analytics with sorted sets and hashes
  • Managing complex caching strategies across multiple layers
  • Supporting real-time streaming and pub/sub messaging

The learning curve was steep, especially understanding vector search and Redis's query syntax, but the performance and capabilities I achieved made every challenge worthwhile. Building NewsHub taught me how Redis can accelerate AI applications and handle complex, real-time workloads that would typically require multiple specialized systems.

NewsHub demonstrates how Redis 8 can power the next generation of AI applications, serving as a complete real-time intelligence platform. The combination of vector search, JSON storage, multi-layer caching, and real-time analytics creates a seamless, intelligent user experience that adapts in real-time to user behavior and preferences.

Comments

Post a Comment

Popular posts from this blog

Step by Step Guide on Converting Text to High-Quality Audio Using an Open Source TTS Model on Hugging Face: Including Detailed Audio File Analysis and Diagnostic Tools in Python

Image
  In this tutorial, we demonstrate a complete end-to-end solution to convert text into audio using an open-source text-to-speech (TTS) model available on Hugging Face. Leveraging the capabilities of the Coqui TTS library, the tutorial walks you through initializing a state-of-the-art TTS model (in our case, “tts_models/en/ljspeech/tacotron2-DDC”), processing your input text, and saving the resulting synthesis as a high-quality WAV audio file. In addition, we integrate Python’s audio processing tools, including the wave module and context managers, to analyze key audio file attributes like duration, sample rate, sample width, and channel configuration. This step-by-step guide is designed to cater to beginners and advanced developers who want to understand how to generate speech from text and perform basic diagnostic analysis on the output. Copy Code !pip install TTS !pip install TTS installs the Coqui TTS library, enabling you to leverage open-source text-to-speech models to convert...
Read more

Aldi hiring over 13K workers nationwide, raising pay

Image
  Discount grocer Aldi   is boosting its staff for the holiday season with plans to hire more than 13,000 store and warehouse workers.  Along with boosting its workforce, the grocer announced that it will continue adding new stores across the U.S. while it simultaneously prepares for the busy shopping period.  To entice applicants, the company also announced that it's raising the national average hourly pay too.  LOOKING FOR A JOB? THESE 5 STATES HAVE MOST COMPETITION The average starting wages for Aldi store and warehouse positions nationwide will jump to $18 and $23 per hour, respectively, based on market and position, according to the company.  The average starting wages for Aldi store and warehouse positions nationwide will jump to $18 and $23 per hour, respectively, based on market and position, according to the company.  (Burak Bir/Anadolu Agency via Getty Images / Getty Images) The company also touted that the company "outranks competitors" in...
Read more

The Case Against AI Tools for Writing

Image
My relationship with Artificial Intelligence is...   complicated.   This is partially because there's a big difference between my personal and professional life... Professionally  -   I rigorously studied Machine Learning in Graduate School (and it was a heavy component of my Master's thesis). Even now, in my 9-5 day job, I am a research contractor who works on studies that employ the use of Machine Learning. Personally  - I like to write silly little stories and poems. So, as someone who is fascinated with both AI and writing, you would think that I would be jumping all over the opportunity to marry my personal and professional lives. But the reality is, I think it's a waste of both the potential of AI and the potential of writers. So if I have not lost you already, let me try and state my case. By  Hitesh Choudhary  on  Unsplash Let's start with some definitions To make sure that everyone is on the same page, I wanted to include a few definition...
Read more