Question

我创建了一些按以下关键字过滤的Twitter流。

TwitterStream twitterStream = getTwitterStreamInstance();
FilterQuery filtre = new FilterQuery();
String[] keywordsArray = { "iphone", "samsung" , "apple", "amazon"};
filtre.track(keywordsArray);
twitterStream.filter(filtre);
twitterStream.addListener(listener);

根据匹配的关键字分隔推文的最佳方式是什么？例如所有匹配“iphone”的推文都应该存储到“IPHONE”表中，所有匹配“samsung”的推文都会存储到“SAMSUNG”表中，依此类推。注意：过滤关键字的数量约为500。

Answer 1

以下是您如何使用StatusListener查询收到的Status个对象：

final Set<String> keywords = new HashSet<String>();
keywords.add("apple");
keywords.add("samsung");
// ...

final StatusListener listener = new StatusAdapter() {
    @Override
    public void onStatus(Status status) {
        final String statusText = status.getText();
        for (String keyword : keywords) {
            if (statusText.contains(keyword)) {
                dao.insert(keyword, statusText);
            }
        }
    }
};

final TwitterStream twitterStream = getTwitterStreamInstance();
final FilterQuery fq = new FilterQuery();
fq.track(keywords.toArray(new String[0]));
twitterStream.addListener(listener);
twitterStream.filter(fq);

我看到DAO的定义如下：

public interface StatusDao {
    void insert(String tableSuffix, Status status);
}

然后，您将拥有与每个关键字对应的数据库表。实现将使用tableSuffix将Status存储在正确的表中，sql大致如下：

INSERT INTO status_$tableSuffix$ VALUES (...)

注意：

如果推文包含＆＃39; apple＆＃39;此实现会将Status插入多个表格中。和＆＃39;三星＆＃39;例如。
此外，这是一个非常简单的实现，您可能需要考虑将插入批量添加到表中...但这取决于您将接收的推文的数量。
如评论中所述，API会在匹配时考虑其他属性，例如网址和嵌入式推文（如果有），因此搜索状态文本以查找关键字匹配可能还不够。

Answer 2

似乎找出推文属于哪个关键字的唯一方法是迭代Status对象的多个属性。以下代码需要具有方法insertTweet(String tweetText, Date createdAt, String keyword)的数据库服务，并且如果找到多个关键字，则每个推文都会多次存储在数据库中。如果在推文文本中找到至少一个关键字，则不会搜索其他属性以获取更多关键字。

// creates a map of the keywords with a compiled pattern, which matches the keyword
private Map<String, Pattern> keywordsMap = new HashMap<>();
private TwitterStream twitterStream;
private DatabaseService databaseService; // implement and add this service

public void start(List<String> keywords) {
    stop(); // stop the streaming first, if it is already running

    if(keywords.size() > 0) {
        for(String keyword : keywords) {
            keywordsMap.put(keyword, Pattern.compile(keyword, Pattern.CASE_INSENSITIVE));
        }

        twitterStream = new TwitterStreamFactory().getInstance();
        StatusListener listener = new StatusListener() {
            @Override
            public void onStatus(Status status) {
                insertTweetWithKeywordIntoDatabase(status);
            }
            /* add the unimplemented methods from the interface */
        };
        twitterStream.addListener(listener);
        FilterQuery filterQuery = new FilterQuery();
        filterQuery.track(keywordsMap.keySet().toArray(new String[keywordsMap.keySet().size()]));
        filterQuery.language(new String[]{"en"});

        twitterStream.filter(filterQuery);
    }
    else {
        System.err.println("Could not start querying because there are no keywords.");
    }
}

public void stop() {
    keywordsMap.clear();
    if(twitterStream != null) {
        twitterStream.shutdown();
    }
}

private void insertTweetWithKeywordIntoDatabase(Status status) {
    // search for keywords in tweet text
    List<String> keywords = getKeywordsFromTweet(status.getText());

    if (keywords.isEmpty()) {
        StringBuffer additionalDataFromTweets = new StringBuffer();

        // get extended urls
        if (status.getURLEntities() != null) {
            for (URLEntity url : status.getURLEntities()) {
                if (url != null && url.getExpandedURL() != null) {
                    additionalDataFromTweets.append(url.getExpandedURL());
                }
            }
        }

        // get retweeted status -> text
        if (status.getRetweetedStatus() != null && status.getRetweetedStatus().getText() != null) {
            additionalDataFromTweets.append(status.getRetweetedStatus().getText());
        }
        // get retweeted status -> quoted status -> text
        if (status.getRetweetedStatus() != null && status.getRetweetedStatus().getQuotedStatus() != null
                && status.getRetweetedStatus().getQuotedStatus().getText() != null) {
            additionalDataFromTweets.append(status.getRetweetedStatus().getQuotedStatus().getText());
        }
        // get retweeted status -> quoted status -> extended urls
        if (status.getRetweetedStatus() != null && status.getRetweetedStatus().getQuotedStatus() != null
                && status.getRetweetedStatus().getQuotedStatus().getURLEntities() != null) {
            for (URLEntity url : status.getRetweetedStatus().getQuotedStatus().getURLEntities()) {
                if (url != null && url.getExpandedURL() != null) {
                    additionalDataFromTweets.append(url.getExpandedURL());
                }
            }
        }

        // get quoted status -> text
        if (status.getQuotedStatus() != null && status.getQuotedStatus().getText() != null) {
            additionalDataFromTweets.append(status.getQuotedStatus().getText());
        }
        // get quoted status -> extended urls
        if (status.getQuotedStatus() != null && status.getQuotedStatus().getURLEntities() != null) {
            for (URLEntity url : status.getQuotedStatus().getURLEntities()) {
                if (url != null && url.getExpandedURL() != null) {
                    additionalDataFromTweets.append(url.getExpandedURL());
                }
            }
        }

        String additionalData = additionalDataFromTweets.toString();
        keywords = getKeywordsFromTweet(additionalData);
    }

    if (keywords.isEmpty()) {
        System.err.println("ERROR: No Keyword found for: " + status.toString());

    } else {
        // insert into database
        for(String keyword : keywords) {
            databaseService.insertTweet(status.getText(), status.getCreatedAt(), keyword);
        }
    }

}

// returns a list of keywords which are found in a tweet
private List<String> getKeywordsFromTweet(String tweet) {
    List<String> result = new ArrayList<>();

    for (String keyword : keywordsMap.keySet()) {
        Pattern p = keywordsMap.get(keyword);
        if (p.matcher(tweet).find()) {
            result.add(keyword);
        }
    }

    return result;
}

Answer 3

好吧，你可以创建一个类似于ArrayList的类，但是这样就可以创建一个ArrayLists数组，称之为TweetList。这个类需要一个插入函数。

然后使用两个for循环来搜索推文并找到包含在普通数组列表中的匹配关键字，然后将它们添加到与关键字ArrayList中关键字索引相匹配的TweetList

 for (int i = 0; i < tweets.length; i++)
           {
               String[] split = tweets[i].split(" ");// split the tweet up
               for (int j = 0; j < split.length; j++)
                   if (keywords.contains(split[j]))//check each word against the keyword list
                       list[keywords.indexOf(j)].insert[tweets[i]];//add the tweet to the tree index that matches index of the keyword
           }

根据匹配的关键字隔离过滤的推文：Twitter4j API

3 个答案:

注意：