AI Trees instead of AI Factories

Solar Powered independent AI Trees

I had an interesting idea that maybe we should make AI Trees in addition to the huge AI Factories (Data Centers).

The concept being that Trees have leaves which use solar energy. That’s the corelation with Trees. The concept is that there will be a completely independent 1 Rack solution with sufficent amount of solar power and limited amount of GPU compute.

I used AI to calculate the power, GPUs, inference, users and Grok AI gave interesting technical specifications of such independent trees.

Please see the discussion below.

Question:

I want to create a Solar powered data center rack which handles AI workloads.

Regarding the technical specs. How many GPUs could a solar powered rack support and how much inference could it do ?

Answer:

To estimate the number of GPUs a solar-powered data center rack can support and the inference throughput it can achieve, we need to analyze the technical specifications provided in the concept paper and make reasonable assumptions based on current hardware and power constraints. Below is a detailed breakdown.

Key Technical Specifications

Power Capacity: 10-20 kW per rack, depending on solar panel and battery configuration.

Compute Performance: Up to 100 TFLOPS per rack for AI workloads (based on GPU configuration).

Cooling Capacity: Supports up to 15 kW of heat dissipation with liquid and passive cooling.

Form Factor: Standard 42U rack (600mm W x 1200mm D x 2000mm H).

Battery Backup: 48-72 hours of operation without sunlight, depending on workload.

Assumptions

GPU Choice: We’ll assume NVIDIA A100 80GB GPUs (SXM form factor), a common choice for AI workloads in 2025, with a TDP of ~400W per GPU and ~70 TFLOPS (FP16 precision) for AI tasks. These are efficient and widely used for both training and inference.

Power Allocation: ~70% of the rack’s power budget is allocated to GPUs, with the remainder for CPUs, memory, storage, networking, and cooling overhead. This is typical for AI-optimized racks.

Inference Workload: Inference throughput depends on model size, batch size, and data type (e.g., FP16). We’ll use a representative model like a large language model (LLM) with ~70B parameters (e.g., similar to LLaMA 3) for estimation.

Cooling Efficiency: The 15 kW cooling capacity is sufficient to handle the heat from the GPUs and other components, as liquid cooling is highly efficient.

Step 1: Estimating GPU Capacity

Let’s calculate how many GPUs the rack can support based on power constraints.

Power Budget for GPUs: Assume 70% of the rack’s power capacity is available for GPUs.

For a 20 kW rack: 20,000 W × 0.7 = 14,000 W (14 kW).

For a 10 kW rack: 10,000 W × 0.7 = 7,000 W (7 kW).

Power per GPU: NVIDIA A100 80GB has a TDP of ~400W. Accounting for minor overhead (e.g., power supply inefficiency), assume ~450W per GPU.

Number of GPUs:

High-end (20 kW rack): 14,000 W ÷ 450 W/GPU ≈ 31 GPUs.

Low-end (10 kW rack): 7,000 W ÷ 450 W/GPU ≈ 15 GPUs.

Physical Space: A 42U rack can accommodate ~4-8 GPU servers (e.g., NVIDIA DGX A100 or similar, each with 8 GPUs, occupying ~8-10U). With 5 servers (40U, leaving space for networking/storage), the rack could fit 40 GPUs physically, but power limits the number to 15-31 GPUs.

Conclusion: The rack can support 15-31 NVIDIA A100 GPUs, depending on the power configuration (10-20 kW).

Step 2: Estimating Inference Throughput

Inference throughput depends on the model, batch size, and GPU performance. Let’s estimate for a 70B-parameter LLM performing inference in FP16 precision.

GPU Performance: Each A100 80GB delivers ~312 TFLOPS for FP16 sparse operations (as per NVIDIA specs). For dense FP16, assume ~70 TFLOPS (aligned with the rack’s 100 TFLOPS estimate for ~15 GPUs).

Model Inference:

A 70B-parameter LLM requires ~140 GB of VRAM in FP16 (2 bytes per parameter). With 80 GB per A100, multiple GPUs are needed for model parallelism or pipeline parallelism.

Assume 4 GPUs per model instance (320 GB total VRAM, sufficient for the model and activations with batch size ~16-32).

Inference Speed:

Based on benchmarks for similar models (e.g., LLaMA 70B on A100s), a single A100 can process ~10-20 tokens/second for inference with batch size 16, depending on optimizations (e.g., TensorRT-LLM).

With 4 GPUs, one model instance can achieve ~30-50 tokens/second (conservative estimate, assuming pipeline parallelism overhead).

Parallel Instances:

20 kW rack (31 GPUs): 31 ÷ 4 ≈ 7 model instances (28 GPUs, with 3 GPUs for other tasks). Total throughput: 7 × 30-50 tokens/second = 210-350 tokens/second.

10 kW rack (15 GPUs): 15 ÷ 4 ≈ 3 model instances (12 GPUs, with 3 GPUs for other tasks). Total throughput: 3 × 30-50 tokens/second = 90-150 tokens/second.

Context for Tokens/Second:

For text generation (e.g., chatbots), 30-50 tokens/second per instance translates to ~1-2 responses/second for 20-50 token outputs (typical for conversational AI).

For batch inference (e.g., text classification), throughput could be higher with larger batch sizes, potentially processing thousands of samples per second.

Step 3: Practical Considerations

Solar Power Stability: The battery backup (48-72 hours) ensures consistent operation, but throughput may be throttled during extended low-sunlight periods to conserve energy.

Cooling Constraints: The 15 kW cooling capacity supports ~31 GPUs (31 × 400W = 12.4 kW heat), confirming feasibility for the high-end configuration.

Workload Variability: Inference throughput varies by model size and task. Smaller models (e.g., 7B parameters) could run on single GPUs, increasing parallel instances and throughput significantly (e.g., 15-31 instances for 20-50 tokens/second each).

Optimizations: Using INT8 quantization or frameworks like vLLM could boost throughput by 1.5-2x, potentially reaching 300-700 tokens/second for the 20 kW rack.

Final Answer

Number of GPUs: The solar-powered rack can support 15-31 NVIDIA A100 80GB GPUs, depending on the power capacity (10-20 kW).

Inference Throughput: For a 70B-parameter LLM in FP16, the rack can achieve 90-150 tokens/second (10 kW, ~3 model instances) to 210-350 tokens/second (20 kW, ~7 model instances). With optimizations or smaller models, throughput could reach 300-700 tokens/second for the high-end configuration.

If you have a specific model or workload in mind (e.g., vision models, smaller LLMs), I can refine the estimate further. Would you like me to explore alternative GPU options or specific AI tasks?

Follow up Question:

Yes tell me possibility for Deepseek V3

Answer:

To evaluate the feasibility of running DeepSeek V3, a 671B parameter Mixture-of-Experts (MoE) language model with 37B active parameters per token, on the proposed solar-powered data center rack for AI workloads, we need to assess its compatibility with the rack’s technical specifications, estimate the number of GPUs required, and calculate the inference throughput. This analysis builds on the previous response, which outlined the rack’s capabilities and estimated GPU support for NVIDIA A100 GPUs running a 70B-parameter LLM. Here, we adapt the calculations for DeepSeek V3, leveraging available information about its architecture and performance.

Recap of Rack Specifications

Power Capacity: 10-20 kW per rack.

Cooling Capacity: Up to 15 kW of heat dissipation (liquid and passive cooling).

Form Factor: 42U standard rack.

Compute Performance: Up to 100 TFLOPS per rack (based on GPU configuration).

Battery Backup: 48-72 hours of operation without sunlight.

DeepSeek V3 Overview

Architecture: Mixture-of-Experts (MoE) with 671B total parameters, 37B active per token, using Multi-head Latent Attention (MLA) and Multi-Token Prediction (MTP).

Performance: Excels in math, coding, and reasoning, with a token generation speed of ~60 tokens/second on optimized hardware.

Hardware Requirements: Recommended deployment on 8 NVIDIA H200 GPUs (141GB VRAM each) for optimal performance, though it supports NVIDIA, AMD, and Huawei Ascend hardware.

Context Window: Up to 128K tokens.

Inference Efficiency: Uses FP8 precision and MoE sparsity, reducing memory and compute needs compared to dense models of similar size.

Step 1: Estimating GPU Requirements

DeepSeek V3’s MoE architecture activates only 37B parameters per token, making its computational and memory requirements closer to a dense 37B-parameter model for inference, rather than the full 671B. Let’s estimate the number of GPUs needed and how many can fit within the rack’s power and cooling constraints.

Memory Requirements

Model Size: For FP16 inference, 37B active parameters require ~74GB VRAM (2 bytes per parameter, excluding activations and KV cache). With MTP modules (14B parameters), total VRAM is ~80-100GB per model instance, depending on batch size and context length.

GPU Choice: Assume NVIDIA H200 GPUs (141GB VRAM, ~3 TFLOPS FP16, TDP ~700W) for compatibility with DeepSeek’s recommendation.

A single H200 can likely host one DeepSeek V3 instance for small batch sizes (e.g., 1-4), but larger batches or long contexts may require 2-4 GPUs for parallelism (tensor or pipeline).

Conservative Estimate: Assume 2 H200 GPUs per model instance to handle activations, KV cache for 128K context, and batch sizes up to 16, requiring ~150-200GB total VRAM.

Power and Cooling Constraints

Power per GPU: H200 TDP is ~700W, but with system overhead (e.g., power supply, networking), assume ~800W per GPU.

Rack Power Budget:

20 kW rack: 70% for GPUs = 14,000 W ÷ 800 W/GPU ≈ 17 GPUs.

10 kW rack: 70% for GPUs = 7,000 W ÷ 800 W/GPU ≈ 8 GPUs.

Cooling: 17 H200 GPUs produce ~17 × 700W = 11.9 kW of heat, within the 15 kW cooling capacity.

Physical Space: A 42U rack can fit ~4-5 servers (e.g., NVIDIA DGX H200, 8 GPUs each, ~8-10U). With 2 servers (16 GPUs), the rack stays within power (14 kW) and cooling limits.

Conclusion: The rack can support 8-17 H200 GPUs, translating to 4-8 DeepSeek V3 instances (assuming 2 GPUs per instance for robust inference).

Step 2: Inference Throughput

DeepSeek V3’s inference throughput depends on the number of model instances, batch size, and optimizations (e.g., FP8, MTP, vLLM). Let’s estimate based on reported performance and rack constraints.

Baseline Performance: DeepSeek V3 achieves ~60 tokens/second on optimized hardware (e.g., 8 H200 GPUs with vLLM, FP8 precision).

For 2 H200 GPUs per instance, assume a conservative ~20-30 tokens/second per instance (due to reduced parallelism but still leveraging MTP and MLA).

Model Instances:

20 kW rack: 17 GPUs ÷ 2 GPUs/instance ≈ 8 instances × 20-30 tokens/second = 160-240 tokens/second.

10 kW rack: 8 GPUs ÷ 2 GPUs/instance ≈ 4 instances × 20-30 tokens/second = 80-120 tokens/second.

Optimizations:

Using vLLM or TensorRT-LLM with FP8 and MTP could boost throughput by ~1.5x, potentially reaching 240-360 tokens/second (20 kW) or 120-180 tokens/second (10 kW).

Larger batch sizes (e.g., 16-32) or shorter contexts (<128K) could further increase throughput.

Context for Tokens/Second:

For conversational tasks (e.g., chatbots), 20-30 tokens/second per instance supports ~1-2 responses/second for 20-50 token outputs.

For batch inference (e.g., text classification, code generation), throughput could reach thousands of samples/second with high batch sizes.

Step 3: Solar Power Feasibility

Power Stability: The rack’s battery backup (48-72 hours) ensures uninterrupted operation during low-sunlight periods, critical for consistent inference.

Energy Efficiency: DeepSeek V3’s MoE architecture and FP8 precision reduce power consumption compared to dense models. For 17 H200 GPUs (~13.6 kW), a 20 kW solar system with battery storage is sufficient, assuming adequate sunlight (e.g., 5-6 kWh/m²/day in sunny regions).

Throttling: During extended low-sunlight periods, the system could reduce active GPUs (e.g., from 17 to 8), lowering throughput to ~80-120 tokens/second to conserve energy.

Step 4: Practical Considerations

Hardware Compatibility: DeepSeek V3 supports NVIDIA GPUs (H200 recommended), AMD, and Huawei Ascend NPUs. The rack’s modular design can accommodate H200-based servers, ensuring compatibility.

Cooling: The 15 kW cooling capacity handles ~11.9 kW heat from 17 H200 GPUs, with liquid cooling optimizing efficiency for high-density AI workloads.

Software Stack: Use vLLM, SGLang, or TensorRT-LLM for inference, leveraging DeepSeek’s MLA and MTP optimizations. SGLang supports multi-node tensor parallelism, ideal for scaling across racks.

Use Cases:

Edge AI: Run DeepSeek V3 for real-time reasoning, coding, or math tasks in remote locations (e.g., smart grids, autonomous systems).

Research: Support academic labs with cost-effective, high-performance inference for NLP, code generation, or theorem proving.

Commercial: Deploy for multilingual chatbots, automated coding, or data analysis with low operational costs due to solar power.

Step 5: Comparison to Previous Estimate

The previous estimate for a 70B-parameter LLM assumed 4 A100 GPUs per instance, yielding 90-350 tokens/second across 3-7 instances. DeepSeek V3, with 37B active parameters, requires fewer GPUs per instance (2 H200s) due to MoE sparsity, allowing more instances (4-8) and comparable throughput (80-360 tokens/second). The H200’s higher VRAM and efficiency, combined with DeepSeek’s optimizations, make it better suited for the rack than the A100 for this model.

Final Answer

The solar-powered data center rack can support 4-8 DeepSeek V3 instances using 8-17 NVIDIA H200 GPUs, depending on the power configuration (10-20 kW). Inference throughput is estimated at 80-180 tokens/second (10 kW) to 160-360 tokens/second (20 kW), with optimizations potentially reaching the higher end. The rack’s solar power, battery backup, and cooling capacity make it feasible for running DeepSeek V3 in sustainable, high-performance AI applications, particularly for math, coding, and reasoning tasks at the edge or in remote settings.

If you want to explore specific tasks (e.g., coding vs. math), alternative GPUs (e.g., AMD), or deployment scenarios, let me know!

Follow up Question regarding the number of users supported by the system.

Tell me how many users can be supported by these numbers of tokens/second

Answer:

To estimate how many users can be supported by the DeepSeek V3 inference throughput on the solar-powered data center rack, we need to consider the token generation rates provided, typical user interaction patterns, and the context of the application. The previous response estimated inference throughput for DeepSeek V3 at 80-180 tokens/second (10 kW rack) to 160-360 tokens/second (20 kW rack), with optimized scenarios reaching the higher end. Below, we’ll calculate the number of supported users based on these rates, focusing on conversational AI as the primary use case, and account for variations in usage patterns.

Step 1: Understanding Token Generation and User Interaction

Tokens per Interaction:

In conversational AI (e.g., chatbots, coding assistants), a typical user query might generate 20-50 tokens for a response (e.g., a short paragraph or code snippet). For DeepSeek V3, optimized for reasoning, coding, and math, responses might be slightly longer for complex tasks (e.g., 50-100 tokens for code or explanations).

Assume an average response length of 50 tokens for general conversational tasks, with a range of 20-100 tokens for sensitivity analysis.

User Interaction Rate:

Active Users: In a chatbot scenario, an active user might send 1 query every 10-30 seconds during a session (e.g., 2-6 queries/minute). For enterprise or research use, assume a more conservative 2 queries/minute (1 query every 30 seconds) to account for thinking/reading time.

Concurrent Users: The number of users supported depends on how many queries the system can handle simultaneously, constrained by the tokens/second throughput.

Latency Expectation: Users expect responses within ~1-2 seconds. DeepSeek V3’s throughput (e.g., 20-30 tokens/second per instance) suggests a single instance can generate a 50-token response in ~1.7-2.5 seconds, which is acceptable but may require batching for multiple users to minimize wait times.

Step 2: Calculating Queries per Second

Tokens per Second:

10 kW rack: 80-180 tokens/second (base) to 120-180 tokens/second (optimized).

20 kW rack: 160-360 tokens/second (base) to 240-360 tokens/second (optimized).

Queries per Second (assuming 50 tokens per response):

10 kW rack:

Base: 80 ÷ 50 = 1.6 queries/second, 180 ÷ 50 = 3.6 queries/second.

Optimized: 120 ÷ 50 = 2.4 queries/second, 180 ÷ 50 = 3.6 queries/second.

20 kW rack:

Base: 160 ÷ 50 = 3.2 queries/second, 360 ÷ 50 = 7.2 queries/second.

Optimized: 240 ÷ 50 = 4.8 queries/second, 360 ÷ 50 = 7.2 queries/second.

Range: The rack can handle 1.6-7.2 queries/second, with the high end reflecting the 20 kW rack’s optimized performance.

Step 3: Estimating Concurrent Users

Queries per User: Assume each user generates 2 queries/minute (1 query every 30 seconds), or 2 ÷ 60 = 0.033 queries/second.

Users Supported (Queries per second ÷ Queries per user):

10 kW rack:

Base: 1.6 ÷ 0.033 ≈ 48 users, 3.6 ÷ 0.033 ≈ 109 users.

Optimized: 2.4 ÷ 0.033 ≈ 73 users, 3.6 ÷ 0.033 ≈ 109 users.

20 kW rack:

Base: 3.2 ÷ 0.033 ≈ 97 users, 7.2 ÷ 0.033 ≈ 218 users.

Optimized: 4.8 ÷ 0.033 ≈ 145 users, 7.2 ÷ 0.033 ≈ 218 users.

Range: The rack can support 48-218 concurrent users, with the 20 kW rack in optimized mode (360 tokens/second) handling up to 218 users at 2 queries/minute.

Step 4: Sensitivity Analysis

Response Length:

20 tokens/response (short answers): Queries/second = 80 ÷ 20 = 4 (low) to 360 ÷ 20 = 18 (high). Users = 4 ÷ 0.033 ≈ 121 users to 18 ÷ 0.033 ≈ 545 users.

100 tokens/response (long answers, e.g., code): Queries/second = 80 ÷ 100 = 0.8 (low) to 360 ÷ 100 = 3.6 (high). Users = 0.8 ÷ 0.033 ≈ 24 users to 3.6 ÷ 0.033 ≈ 109 users.

Query Rate:

4 queries/minute (1 query every 15 seconds, heavy usage): Queries/second per user = 4 ÷ 60 = 0.067. Users = 1.6 ÷ 0.067 ≈ 24 users to 7.2 ÷ 0.067 ≈ 107 users.

1 query/minute (light usage, e.g., occasional queries): Queries/second per user = 1 ÷ 60 = 0.017. Users = 1.6 ÷ 0.017 ≈ 94 users to 7.2 ÷ 0.017 ≈ 423 users.

Range Across Scenarios: Depending on response length and query rate, the rack supports 24-545 concurrent users, with realistic scenarios (50 tokens, 2 queries/minute) yielding 48-218 users.

Step 5: Practical Considerations

Batching and Latency: To support multiple users, queries are batched (e.g., batch size 16-32 per model instance). With 4-8 DeepSeek V3 instances (10-20 kW rack), the system can process multiple queries in parallel, keeping latency below 2 seconds. For example, at 360 tokens/second (7.2 queries/second), batching 16 queries across 8 instances ensures ~1-second responses.

Workload Variability:

Conversational AI: The 48-218 user range is suitable for enterprise chatbots, customer support, or coding assistants, where users expect quick responses.

Batch Inference: For non-real-time tasks (e.g., bulk code generation, data analysis), the system could process thousands of queries/hour by maximizing batch sizes, supporting far more “users” (e.g., API calls) but not concurrently.

Solar Power Stability: The 48-72 hour battery backup ensures consistent throughput during low-sunlight periods. If power is throttled (e.g., reducing to 8 GPUs), user capacity drops to ~48-109 users (10 kW performance).

Use Case Context:

Edge Deployments: 48-218 users is sufficient for small communities, remote research teams, or localized AI services (e.g., smart cities, disaster response).

Enterprise: For larger organizations, multiple racks could scale to thousands of users, sharing solar infrastructure.

Optimizations: Using vLLM, SGLang, or FP8 precision maximizes throughput. Dynamic load balancing across instances can further optimize user support during peak usage.

Step 6: Comparison to Industry Benchmarks

Typical cloud-based AI services (e.g., ChatGPT, Claude) handle thousands of users by scaling across massive GPU clusters. The solar-powered rack, with 8-17 H200 GPUs, is designed for edge or small-scale deployments, supporting tens to hundreds of users, which aligns with use cases like remote research labs or enterprise-internal AI tools.

DeepSeek V3’s MoE efficiency (37B active parameters) allows it to outperform denser models (e.g., 70B LLMs) on the same hardware, making the 48-218 user range competitive for its class.

Final Answer

The solar-powered data center rack, running DeepSeek V3 with 80-360 tokens/second, can support 48-218 concurrent users for conversational AI tasks, assuming 50-token responses and 2 queries/minute per user. With shorter responses (20 tokens) or lighter usage (1 query/minute), this could increase to 94-545 users. For heavier usage (4 queries/minute) or longer responses (100 tokens), the range drops to 24-109 users. The 20 kW rack in optimized mode (360 tokens/second) achieves the high end (218 users), suitable for edge deployments, research labs, or small enterprises, with battery backup ensuring reliability.

End of AI Discussion

This offers an interesting considering for inference.

Decentralized Inference

AI Trees can be used to offer decentralized inference instead of AI Factories offering centralized inference.

Thank you,

Habib