Voice of Voiceless是一款基于 Streamlit 的尖端应用程序，旨在通过超快速的实时语音转录、情绪音调检测和情感分析，弥合聋人和听力障碍人士之间的沟通鸿沟。该应用程序专为 AssemblyAI 语音代理挑战赛而开发，展示了 300 毫秒以下语音处理速度在对无障碍访问要求极高的场景中的巨大潜力。
该应用程序不仅仅是一个转录工具，它还是一个全面的沟通助手，不仅提供关于所说内容的视觉反馈，还提供关于如何说话的视觉反馈，从而为听不到声音提示的用户创造更丰富的对话理解。
本次提交的作品针对实时语音性能类别，重点关注以下方面：
该应用程序提供了一套全面的、以辅助功能为中心的特性：
“无声者之声”应用程序可使用 Streamlit 在本地运行。该界面提供直观、注重无障碍的用户体验，并具备实时更新和全面的视觉反馈系统。
主界面 - 实时转录
主界面采用简洁、高对比度的设计，配有大而易读的文本和清晰的连接状态和性能指标视觉指示器。
辅助功能控制面板
侧边栏提供全面的辅助功能控制选项，包括：
情感和语气分析：
实时显示情绪智能，包括：
性能仪表盘
实时显示性能指标：
该应用程序演示了几个关键场景：
mohamednizzad /无声之声
🏆 AssemblyAI 语音代理挑战赛参赛作品 - 实时语音性能类别
VoiceAccess 是一款基于 Streamlit 的尖端应用程序，旨在通过提供超快速的实时语音转录、音调检测和情感分析来帮助聋人和听力障碍人士。它采用 AssemblyAI 的通用流式 API 构建，可为关键的辅助功能应用提供低于 300 毫秒的延迟。
本项目致力于利用 AssemblyAI 的通用流媒体技术，打造速度最快、响应最灵敏的语音体验。该技术专为对可访问性要求极高的用例而设计，在这些用例中，低于 300 毫秒的延迟至关重要。
完整的源代码以及详尽的文档、安装指南和示例配置均已提供。该代码库包含：
Voice of Voiceless 采用复杂的多线程架构，旨在实现最佳的实时性能：
# Core application structure
class VoiceAccessApp:
def __init__(self):
self.audio_processor = AudioProcessor()
self.transcription_service = TranscriptionService()
self.ui_components = UIComponents()
self.accessibility = AccessibilityFeatures()
self.performance_monitor = PerformanceMonitor()
该应用程序将功能划分为五个主要模块：
VoiceAccess 的核心在于它与 AssemblyAI 的通用流媒体 API 的精妙集成：
class TranscriptionService:
def __init__(self):
self.api_key = os.getenv('ASSEMBLYAI_API_KEY')
aai.settings.api_key = self.api_key
# Configure for optimal performance
self.config = {
'sample_rate': 16000,
'enable_speaker_diarization': True,
'enable_sentiment_analysis': True,
'confidence_threshold': 0.7
}
def connect(self) -> bool:
"""Connect to AssemblyAI real-time transcription"""
self.transcriber = aai.RealtimeTranscriber(
sample_rate=self.config['sample_rate'],
on_data=self._on_data,
on_error=self._on_error,
)
self.transcriber.connect()
return True
def _on_data(self, transcript: aai.RealtimeTranscript):
"""Handle real-time transcription with latency tracking"""
request_start = time.time()
result = TranscriptionResult(
text=transcript.text,
confidence=getattr(transcript, 'confidence', 0.0),
speaker=getattr(transcript, 'speaker', None),
timestamp=datetime.now(),
is_final=not transcript.partial
)
# Calculate and track latency
latency = (time.time() - request_start) * 1000
self.total_latency += latency
# Trigger callbacks for UI updates
for callback in self.callbacks:
callback(result)
音频处理流程经过优化，在保持高品质的同时，最大限度地降低了延迟：
class AudioProcessor:
def __init__(self, config: Optional[AudioConfig] = None):
self.config = config or AudioConfig()
self.audio_queue = queue.Queue(maxsize=100)
def _audio_callback(self, indata, frames, time, status):
"""sounddevice callback optimized for low latency"""
if status:
logger.warning(f"Audio callback status: {status}")
try:
audio_bytes = indata.tobytes()
if not self.audio_queue.full():
self.audio_queue.put(audio_bytes, block=False)
self.total_chunks += 1
else:
self.dropped_chunks += 1
except queue.Full:
self.dropped_chunks += 1
def _preprocess_audio(self, audio_data: bytes) -> bytes:
"""Real-time audio preprocessing for optimal recognition"""
audio_array = np.frombuffer(audio_data, dtype=np.int16)
# Noise gate for clarity
threshold = np.max(np.abs(audio_array)) * 0.1
audio_array = np.where(np.abs(audio_array) < threshold, 0, audio_array)
# Normalize for consistent levels
if np.max(np.abs(audio_array)) > 0:
audio_array = audio_array / np.max(np.abs(audio_array)) * 32767
audio_array = audio_array.astype(np.int16)
return audio_array.tobytes()
除了文字转录之外，VoiceAccess 还实现了先进的音频智能：
def _extract_sentiment(self, transcript) -> Dict[str, Any]:
"""Real-time sentiment analysis with confidence scoring"""
text = transcript.text.lower()
positive_words = ['good', 'great', 'excellent', 'happy', 'love', 'amazing']
negative_words = ['bad', 'terrible', 'awful', 'hate', 'sad', 'angry']
positive_count = sum(1 for word in positive_words if word in text)
negative_count = sum(1 for word in negative_words if word in text)
if positive_count > negative_count:
sentiment_score = min(0.8, positive_count * 0.3)
sentiment_label = 'positive'
elif negative_count > positive_count:
sentiment_score = max(-0.8, -negative_count * 0.3)
sentiment_label = 'negative'
else:
sentiment_score = 0.0
sentiment_label = 'neutral'
return {
'label': sentiment_label,
'score': sentiment_score,
'confidence': 0.75
}
def _detect_tone(self, text: str) -> Dict[str, Any]:
"""Multi-dimensional tone detection"""
tone_patterns = {
'excited': ['!', 'wow', 'amazing', 'incredible', 'fantastic'],
'calm': ['okay', 'fine', 'sure', 'alright', 'peaceful'],
'angry': ['damn', 'hell', 'angry', 'mad', 'furious'],
'sad': ['sad', 'depressed', 'down', 'unhappy', 'crying'],
'happy': ['happy', 'joy', 'cheerful', 'glad', 'delighted']
}
tone_scores = {}
for tone, patterns in tone_patterns.items():
score = sum(1 for pattern in patterns if pattern in text.lower())
tone_scores[tone] = score
max_tone = max(tone_scores.items(), key=lambda x: x[1])
return {
'tone': max_tone[0] if max_tone[1] > 0 else 'neutral',
'confidence': min(0.9, max_tone[1] * 0.3),
'scores': tone_scores
}
VoiceAccess实现了全面的性能监控和优化：
class PerformanceMonitor:
def __init__(self):
self.thresholds = {
'max_latency_ms': 300,
'max_cpu_percent': 80.0,
'max_memory_percent': 85.0,
'min_accuracy': 0.85
}
def _check_performance_alerts(self, metrics: PerformanceMetrics):
"""Real-time performance monitoring with alerts"""
if metrics.latency_ms > self.thresholds['max_latency_ms']:
self._add_alert(
'high_latency',
f"High latency detected: {metrics.latency_ms:.0f}ms",
'warning'
)
if metrics.cpu_percent > self.thresholds['max_cpu_percent']:
self._add_alert(
'high_cpu',
f"High CPU usage: {metrics.cpu_percent:.1f}%",
'warning'
)
def _calculate_performance_score(self, metrics: List[PerformanceMetrics]) -> float:
"""Comprehensive performance scoring algorithm"""
scores = []
# Latency score (lower is better)
latencies = [m.latency_ms for m in metrics if m.latency_ms > 0]
if latencies:
avg_latency = sum(latencies) / len(latencies)
latency_score = max(0, 100 - (avg_latency / self.thresholds['max_latency_ms']) * 100)
scores.append(latency_score)
return sum(scores) / len(scores) if scores else 0.0
VoiceAccess从一开始就将无障碍功能作为首要考虑因素，而不是事后才考虑的：
class AccessibilityFeatures:
def __init__(self):
# WCAG 2.1 AA compliant color schemes
self.high_contrast_colors = {
'background': '#000000',
'text': '#ffffff',
'primary': '#ffffff',
'success': '#00ff00',
'warning': '#ffff00',
'error': '#ff0000'
}
def validate_color_contrast(self, foreground: str, background: str) -> Dict[str, Any]:
"""WCAG 2.1 color contrast validation"""
contrast_ratio = self._calculate_contrast_ratio(foreground, background)
return {
'contrast_ratio': contrast_ratio,
'aa_normal': contrast_ratio >= 4.5,
'aa_large': contrast_ratio >= 3.0,
'aaa_normal': contrast_ratio >= 7.0,
'wcag_level': 'AAA' if contrast_ratio >= 7.0 else 'AA' if contrast_ratio >= 4.5 else 'Fail'
}
该应用程序提供全面的视觉辅助功能选项：
完整的键盘操作功能确保即使无法使用鼠标的用户也能使用该应用程序：
def create_focus_management(self):
"""Comprehensive keyboard navigation implementation"""
focus_script = """
document.addEventListener('keydown', function(e) {
if (e.target.tagName !== 'INPUT' && e.target.tagName !== 'TEXTAREA') {
switch(e.key.toLowerCase()) {
case ' ':
// Space for start/stop recording
const recordButton = document.querySelector('[data-testid="baseButton-secondary"]');
if (recordButton) {
recordButton.click();
e.preventDefault();
}
break;
case 's':
// S for settings panel
const settingsSection = document.querySelector('.stSidebar');
if (settingsSection) {
settingsSection.scrollIntoView();
e.preventDefault();
}
break;
}
}
});
"""
VoiceAccess 通过多种优化策略，始终能够实现低于 300 毫秒的转录延迟：
性能基准测试结果显示：
该应用程序设计轻巧高效：
def get_optimization_recommendations(self) -> List[str]:
"""Dynamic performance optimization suggestions"""
recommendations = []
if avg_latency > self.thresholds['max_latency_ms']:
recommendations.append("Reduce audio chunk size to improve latency")
recommendations.append("Check network connection quality")
if avg_cpu > self.thresholds['max_cpu_percent']:
recommendations.append("Close unnecessary applications to reduce CPU load")
recommendations.append("Consider reducing audio quality settings")
return recommendations
全面的性能监控能够深入了解系统行为：
VoiceAccess率先采用了一种综合性的多模态反馈方法：
def render_transcript_display(self, transcripts: List[Dict], accessibility_settings: Dict):
"""Multi-modal transcript display with rich visual feedback"""
for transcript in transcripts:
confidence_color = "#28a745" if confidence > 0.8 else "#ffc107" if confidence > 0.6 else "#dc3545"
transcript_html = f"""
<div style="
background-color: {'#333333' if high_contrast else '#f8f9fa'};
border-left: 4px solid {confidence_color};
padding: 15px;
margin: 10px 0;
">
<div class="speaker-info">
<strong>{speaker}</strong> • {timestamp} •
<span style="color: {confidence_color}">
{confidence:.1%} confidence
</span>
</div>
<div class="transcript-text">{text}</div>
</div>
"""
界面会根据用户需求和偏好动态调整：
完善的错误处理机制确保持续运行：
def _reconnect(self):
"""Intelligent reconnection with exponential backoff"""
max_retries = 3
retry_delay = 2
for attempt in range(max_retries):
logger.info(f"Reconnection attempt {attempt + 1}/{max_retries}")
self.disconnect()
time.sleep(retry_delay)
if self.connect():
logger.info("Reconnection successful")
return
retry_delay *= 2 # Exponential backoff
logger.error("Failed to reconnect after maximum retries")
VoiceAccess 提供多种安装途径，以适应不同的系统配置：
python install_dependencies.py
pip install -r requirements-minimal.txt
pip install streamlit assemblyai sounddevice numpy python-dotenv pandas plotly psutil requests
考虑到在 Windows 系统上安装 Python 包的挑战，VoiceAccess 包含以下内容：
对于无法安装音频库的系统，VoiceAccess 提供完整的模拟模式：
class FallbackAudioProcessor:
"""Simulation mode for testing without audio hardware"""
def _generate_mock_audio(self) -> bytes:
"""Generate realistic mock audio data"""
samples = np.random.randint(-1000, 1000, self.config.chunk_size, dtype=np.int16)
t = np.linspace(0, 1, self.config.chunk_size)
sine_wave = (np.sin(2 * np.pi * 440 * t) * 500).astype(np.int16)
mixed = (samples * 0.3 + sine_wave * 0.7).astype(np.int16)
return mixed.tobytes()
这样即使没有可用的音频输入，也能演示和测试所有应用程序功能。
VoiceAccess 满足了现实世界中无障碍访问的关键需求：
该应用程序的开源特性和全面的文档使其具备以下优势：
计划改进措施包括：
VoiceAccess 项目代表着在使每个人都能进行实时通信方面迈出了重要一步，它展示了如何利用尖端人工智能技术创造有意义的社会影响，同时在性能和可访问性方面实现卓越的技术水平。
文章来源：https://dev.to/mohamednizzad/voice-of-voiceless-enabling-the-voiceless-to-communicate-1ejp