Finisky Garden

Nowadays, many popular apps incorporate social network features, such as Twitter, WhatsApp, and Facebook. These platforms need to scale to accommodate billions of users (graph nodes), which is no small feat. Building and maintaining a scalable social network infrastructure requires careful planning and strategic data modeling. In fact, specialized social networking applications like Facebook have dedicated teams focusing solely on optimizing their performance to the highest level. However, for smaller apps or startup projects looking to add social networking capabilities, creating a full team to handle such architecture is often impractical and unnecessary.

So, is it possible to build a high-performance, scalable social network using the right data modeling and storage solutions? The answer is yes. Early versions of Facebook used MySQL as the underlying storage to construct their social network, but today we have more advanced and efficient storage options available: MongoDB.

如今许多App都涉及社交网络，如 Twitter、WhatsApp 和 Facebook。这些平台必须扩展以处理数十亿用户（图节点），这并非易事。构建和维护一个可扩展的社交网络基础设施需要仔细的规划和战略性的数据建模。实际上，像Facebook这样专业的社交网络应用有专门的团队来做这块内容，对其性能进行极致的优化。但对于许多希望加入社交网络功能的小型App，如一个创业公司项目，建立一个团队来做这样的架构显然是不现实也没有必要的。

那么，利用合适的数据建模和存储能否构建一个高性能易扩展的社交网络？答案是肯定的。早期的Facebook使用mysql作为底层存储来构建社交网络，但今天我们可以有更好更高效的存储选择：MongoDB。

在conda环境中升级软件包后，talib无法接受DataFrame作为输入，错误信息如下所示：TypeError: Argument 'xxx' has incorrect type (expected numpy.ndarray, got DataFrame)：

Traceback (most recent call last):
  File "/data/1.py", line 7, in <module>
    df['SMA_5'] = ta.SMA(df['Close'], timeperiod=5)
  File "/data/miniconda3/envs/a/lib/python3.10/site-packages/talib/__init__.py", line 64, in wrapper
    result = func(*_args, **_kwds)
TypeError: Argument 'real' has incorrect type (expected numpy.ndarray, got DataFrame)

大多数网络搜索结果都具有误导性，比如将df转换为np数组。由于在更新软件包之前代码能够正常运行，因此问题应为软件包不兼容的问题。

After upgrading packages in a conda env, talib cannot accept dataframe as input, the error message looks like TypeError: Argument 'xxx' has incorrect type (expected numpy.ndarray, got DataFrame):

Traceback (most recent call last):
  File "/data/1.py", line 7, in <module>
    df['SMA_5'] = ta.SMA(df['Close'], timeperiod=5)
  File "/data/miniconda3/envs/a/lib/python3.10/site-packages/talib/__init__.py", line 64, in wrapper
    result = func(*_args, **_kwds)
TypeError: Argument 'real' has incorrect type (expected numpy.ndarray, got DataFrame)

Most of the web search results are misleading, like changing the df into np array. Since the code works before updating packages, the problem should be package incompatibility issue.

Today, I encountered a strange issue in Windows 11 where the D drive was visible in the Disk Management tool but not in File Explorer. I searched online for many solutions, such as updating drivers in Device Manager, disabling and re-enabling the device, using diskpart to delete and recreate the partition, changing the volume label, changing the drive letter, etc., but none worked.

Problem Description

• A new D drive, visible in diskmgmt.msc Disk Management, everything seemed normal. It could even be accessed in File Explorer (though the drive wasn’t displayed in the left sidebar).
• The D drive could be used normally, such as via the command line.
• Changing the drive letter to “E” or another letter made it visible in File Explorer, but switching it back to “D” caused it to disappear again.

今天遇到Win11中 D 盘在磁盘管理工具中显示可用，但在文件管理器中却不可见的诡异情况。网上搜了许多方案，如在设备管理器中更新驱动，禁用再启用设备，用diskpart重新删除新建分区，改卷标改盘符等等都不好使。

问题描述

• 新建D盘，在diskmgmt.msc磁盘管理器中可见，一切正常。甚至能打开文件浏览器（只是左栏不显示磁盘）
• D盘可正常使用，如在命令行中使用
• 修改盘符为“E”或其他盘符，文件浏览器中就可见了，但改回“D”又会消失

Use az cli to query multiple fields of VM information. Here we need to use JMESPath language to implement it.

Typically, we will use az vm show to get the detailed VM information:

$ az vm show -g Linux -n alpha -d -o table
Name    ResourceGroup    PowerState    PublicIps     Fqdns    Location    Zones
------  ---------------  ------------  ------------  -------  ----------  -------
alpha   Linux            VM running    11.1.111.111           eastasia    1

搜索系统的评估和调优很大程度上依赖于相关性标签——这些标签标注了某个文档对特定搜索和搜索者是否有用。理想情况下，这些标签来自真实的搜索用户，但要大规模收集这些数据非常困难，所以典型的实验依赖于第三方标注人员，但他们也可能产生不准确的标注。标注质量一般通过持续的审核、培训和监控来管理。

微软（Bing搜索组）在SIGIR'24提出了一种“反其道而行之”的方法：从真实的用户获取反馈，并利用这些反馈来选择一个与之相符的LLM及其提示词，然后令该LLM大规模地产生标签。实验表明，LLM的准确性与人工标注者相当，并且在找到最佳系统和最难的查询方面同样有用。

[SIGIR2024] # Large Language Models can Accurately Predict Searcher Preferences

大语言模型在各种与语言相关的任务中表现出了显著的零样本泛化能力，包括搜索引擎。然而，现有的工作主要利用LLM的生成能力进行信息检索，而不是直接进行段落排序。这篇EMNLP2023的论文(Outstanding Paper)研究了LLM是否擅长搜索排序的问题。

# Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents

之前写过深度检索模型的介绍：# 深度文本检索模型：DPR, PolyEncoders, DCBERT, ColBERT，今天来看看DeepMind在NeurIPS 2024上的文章，对多向量检索模型（Multi-Vector Retrieval）ColBERT进行了改进：

Rethinking the Role of Token Retrieval in Multi-Vector Retrieval

多向量检索模型由于使Query与Doc进行词元级别的交互，因此在许多信息检索基准测试中达到了SOTA。然而，其非线性评分函数无法扩展到数百万个文档，这就需要一个三阶段的推理过程：通过词元检索检索初始候选，访问所有词元向量，并对初始候选文档进行评分。非线性评分函数应用于每个候选文档的所有词元向量，使得推理过程复杂且缓慢。XTR 引入了新的目标函数，鼓励模型首先检索最重要的文档词元，对词元检索的改进使得 XTR 可以仅使用检索到的词元来对候选文档排序，而不是文档中的所有词元，因此其成本比 ColBERT 低两到三个数量级。在流行的 BEIR 基准测试中，XTR 在没有任何蒸馏的情况下，将 NDCG@10 提升了 2.8。

主要改进点：

• 仅使用检索到的doc token而非全部doc token进行相似度计算
• 解决了检索训练和推理之间的gap