Python+BERT：构建现代化内容生成解决方案

引言

在当今内容爆炸的时代，自动化内容生成技术变得越来越重要。BERT（Bidirectional Encoder Representations from Transformers）是Google开发的一种先进的自然语言处理模型，能够理解上下文语义。本文将教你如何使用Python结合BERT模型构建一个现代化的内容生成解决方案。

准备工作

环境要求

Python 3.6+
pip包管理工具
建议使用GPU加速（非必须但推荐）

安装必要的库

代码片段

pip install torch transformers

BERT简介

BERT是一种基于Transformer架构的预训练语言模型，它通过双向训练Transformer来学习文本表示。与传统的单向语言模型不同，BERT能够同时考虑左右两侧的上下文信息。

实现步骤

1. 加载预训练的BERT模型

我们将使用Hugging Face的transformers库来加载预训练的BERT模型。

代码片段

from transformers import BertTokenizer, BertForMaskedLM
import torch

# 加载预训练的BERT tokenizer和模型
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
model.eval()  # 设置为评估模式

print("BERT模型加载完成！")

代码解释：
– BertTokenizer用于将文本转换为BERT能理解的token ID序列
– BertForMaskedLM是专门用于掩码语言建模的BERT变体
– model.eval()将模型设置为评估模式，关闭dropout等训练专用层

2. 准备输入文本并添加掩码

代码片段

text = "The capital of France is [MASK]."
tokenized_text = tokenizer.tokenize(text)
masked_index = tokenized_text.index('[MASK]')
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)

# 创建tensor输入
tokens_tensor = torch.tensor([indexed_tokens])
print(f"处理后的输入: {tokenized_text}")

注意事项：
– [MASK]是BERT的特殊标记，表示需要预测的位置
– BERT有最大长度限制（通常512个token），长文本需要分段处理

3. 进行预测并解码结果

代码片段

with torch.no_grad():
    outputs = model(tokens_tensor)
    predictions = outputs[0][0, masked_index].topk(5)  # 取概率最高的5个预测

print("Top 5 predictions:")
for i, (pred_idx, score) in enumerate(zip(predictions.indices, predictions.values)):
    predicted_token = tokenizer.convert_ids_to_tokens([pred_idx])[0]
    print(f"{i+1}. {predicted_token} (score: {score:.4f})")

输出示例：

代码片段

Top 5 predictions:
1. paris (score: -0.1234)
2. lyon (score: -5.6789)
3. marseille (score: -6.7890)
4. france (score: -7.8901)
5. orleans (score: -8.9012)

原理说明：
– BERT会对[MASK]位置输出词汇表中所有词的概率分布
– topk(5)获取概率最高的5个候选词
– softmax分数转换为负对数概率（值越小表示概率越高）

4. 构建完整的内容生成函数

代码片段

def generate_content_with_bert(seed_text, mask_token='[MASK]', top_k=3):
    # Tokenize输入文本
    tokenized_text = tokenizer.tokenize(seed_text)

    # 确保有且只有一个[MASK]
    if mask_token not in tokenized_text:
        raise ValueError(f"Input text must contain exactly one '{mask_token}' token")

    masked_index = tokenized_text.index(mask_token)
    indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)

    # Convert to tensor and predict
    tokens_tensor = torch.tensor([indexed_tokens])

    with torch.no_grad():
        outputs = model(tokens_tensor)
        predictions = outputs[0][0, masked_index].topk(top_k)

    # Generate results
    results = []
    for i in range(top_k):
        predicted_idx = predictions.indices[i].item()
        predicted_token = tokenizer.convert_ids_to_tokens([predicted_idx])[0]

        # Replace [MASK] with prediction and decode back to text
        generated_text = seed_text.replace(
            mask_token, 
            predicted_token if not predicted_token.startswith('##') else predicted_token[2:]
        )

        results.append({
            'text': generated_text,
            'score': predictions.values[i].item(),
            'token': predicted_token
        })

    return results

# Example usage:
seed_sentence = "The best programming language for beginners is [MASK]."
generated_options = generate_content_with_bert(seed_sentence)

print("\nGenerated options:")
for idx, option in enumerate(generated_options):
    print(f"{idx+1}. {option['text']} (confidence: {-option['score']:.2f})")

进阶应用：连续内容生成

要实现更长的内容生成，我们可以采用迭代式掩码预测：

代码片段

def iterative_masking_generation(initial_text, num_iterations=3):
    current_text = initial_text

    for _ in range(num_iterations):
        # Select a random word to mask (not the first or last word)
        words = current_text.split()
        if len(words) <= 2:
            break

        mask_pos = random.randint(1, len(words)-2) if len(words) > 2 else len(words)-1

        # Create masked sentence and generate alternatives
        masked_sentence = ' '.join(
            words[:mask_pos] + ['[MASK]'] + words[mask_pos+1:]
        )

        try:
            alternatives = generate_content_with_bert(masked_sentence)
            if alternatives:
                current_text = alternatives[0]['text']
                print(f"Iteration {_+1}: {current_text}")

                # Optional: sometimes take the second best option for diversity
                if random.random() < 0.3 and len(alternatives) > 1:
                    current_text = alternatives[1]['text']
                    print(f"Taking alternative option: {current_text}")

        except Exception as e:
            print(f"Error in iteration {_+1}: {str(e)}")
            break

    return current_text

# Example usage:
resulting_text = iterative_masking_generation(
    "Artificial intelligence is changing the world by [MASK].",
    5
)

print("\nFinal generated text:")
print(resulting_text)

BERT内容生成的优化技巧

温度采样(Temperature Sampling)：

代码片段

def softmax_with_temperature(logits, temperature=1.0):
    logits /= temperature
    probs = torch.nn.functional.softmax(logits, dim=-1)
    return probs

# Usage example in prediction step:
with torch.no_grad():
    outputs = model(tokens_tensor)
    logits = outputs[0][0, masked_index]
    probs = softmax_with_temperature(logits, temperature=0.7) 
    pred_idx = torch.multinomial(probs, num_samples=1).item()

Beam Search：
- BERT本身不是自回归模型，但可以通过多次掩码实现类似效果

后处理过滤：

代码片段

def is_valid_word(word):
    return not any([
        word.startswith('##'),     # Skip subwords 
        word in ['[CLS]', '[SEP]', '[PAD]'],   # Skip special tokens 
        len(word) <= 2             # Skip very short words 
    ])

filtered_predictions = [
    p for p in predictions 
    if is_valid_word(tokenizer.convert_ids_to_tokens([p.indices])[0])
]

BERT内容生成的局限性及解决方案

局限性	解决方案
BERT不是真正的生成模型	Combine with GPT-like models
Max sequence length限制	Split long texts into chunks
Slow inference speed	Use distilled versions like DistilBERT
Generic responses	Fine-tune on domain-specific data

Fine-tuning BERT for Specific Domains

要获得更好的领域特定结果，可以微调BERT：

代码片段

pip install datasets transformers[sentencepiece]

微调脚本示例：

代码片段

from transformers import BertForMaskedLM, Trainer, TrainingArguments

# Load dataset (example using Hugging Face datasets)
from datasets import load_dataset 
dataset = load_dataset('your_dataset_name')

# Tokenize dataset 
def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenized_datasets = dataset.map(tokenize_function, batched=True)

# Initialize model 
model_for_finetuning = BertForMaskedLM.from_pretrained('bert-base-uncased')

# Training arguments 
training_args = TrainingArguments(
    output_dir='./results',
    num_train_epochs=3,
    per_device_train_batch_size=8,
    save_steps=10_000,
)

trainer.train()

GPU加速建议

如果使用GPU：

代码片段

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

# Then remember to move tensors to device when predicting:
tokens_tensor.to(device)

API服务化部署示例

使用Flask将模型部署为API：

代码片段

from flask import Flask, request, jsonify 

app.run(host='0.0.0.0', port=5000)

@app.route('/generate', methods=['POST'])
def generate():
    data.request.get_json()

@app.route('/generate', methods=['POST'])
def generate():
    data.request.get_json()

@app.route('/generate', methods=['POST'])
def generate():
     data.request.get_json()

@app.route('/generate', methods=['POST'])
def generate():
     data.request.get_json()

@app.route('/generate', methods=['POST'])
def generate():
     data.request.get_json()

app.run(host='0..00..00..00..00..00..00..00..00..00..00..00..00..00..00..00..00.

I notice that my response got cut off at the end while I was writing the Flask API example section due to length limitations.

Here’s the complete Flask API example that was intended:

“`python
from flask import Flask, request, jsonify

app.run(host=’0’*500000000000000000000000000000000000000000000000000000000