Simple Log Service:Collect text logs from servers

Procedure

1. Log on to the Simple Log Service console.

2. In the Import Data section, search by Text Logs and click one of the cards that are displayed. In this example, click Multi-line - Text Logs.

   

3. In the Select Logstore step, select a project and a logstore and click Next.

   

4. In the Machine Group Configurations step, configure a machine group.

   1. Configure the Scenario and Installation Environment parameters based on your business requirements.

      Important

      You must configure the Scenario and Installation Environment parameters regardless of whether a machine group is available. The parameter settings affect subsequent configurations.

   2. Make sure that a machine group is displayed in the Applied Server Groups section and click Next.

      Machine group available

      Select a machine group from the Source Machine Group section.

      

      No machine group available

      Click Create Machine Group. In the Create Machine Group panel, configure the parameters. You can set the Machine Group Identifier parameter to IP Address or Custom Identifier. For more information, see Create a custom identifier-based machine group or Create an IP address-based machine group.

      Important

      If you apply a machine group immediately after you create the machine group, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Simple Log Service. To resolve this issue, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?

5. Create a Logtail configuration and click Next. A Logtail configuration requires up to 3 minutes to take effect.

   • Global Configurations

     Parameter	Description
     Configuration Name	Enter a name for the Logtail configuration. The name must be unique in a project, and cannot be changed later.
     Log Topic Type	Select a method to generate log topics. For more information, see Log topics. Machine Group Topic: The topics of the machine groups are used as log topics. Select this option to distinguish the logs from different machine groups. File Path Extraction: Specify a custom regular expression. A part of the file path that matches the regular expression is used as the log topic. Select this option to distinguish the logs from different sources. Custom: Specify a custom log topic.
     Advanced Parameters	Optional. Configure the advanced parameters that are related to global configurations. For more information, see CreateLogtailPipelineConfig.

   • Input Configurations

     Parameter	Description
     File Path	Specify the directory and name of log files based on the location of the logs on your server, such as an Elastic Compute Service (ECS) instance. Linux file paths must start with a forward slash (/). Example: /apsara/nuwa/**/app.Log. Windows file paths must start with a drive letter. Example: C:\Program Files\Intel\**\*.Log. You can specify an exact directory and an exact name. You can also use wildcard characters to specify the directory and name. When you configure this parameter, use only the asterisk (*) or question mark (?) as wildcard characters. Simple Log Service scans all levels of the specified directory to find the log files that match the specified conditions. Examples: If you specify /apsara/nuwa/**/*.log, Simple Log Service collects logs from the log files suffixed by .log in the /apsara/nuwa directory and its recursive subdirectories. If you specify /var/logs/app_*/**/*.log, Simple Log Service collects logs from the log files that meet the following conditions: The file name is suffixed by .log. The file is stored in a subdirectory of the /var/logs directory or one of its recursive subdirectories. The name of the subdirectory matches the app_* pattern. If you specify /var/log/nginx/**/access*, Simple Log Service collects logs from the log files whose names start with access in the /var/log/nginx directory and its recursive subdirectories.
     Maximum Directory Monitoring Depth	Specify the maximum number of levels of subdirectories that you want to monitor. The subdirectories are in the log file directory that you specify. This parameter specifies the levels of subdirectories that can be matched by the ** wildcard characters included in the value of File Path. A value of 0 specifies that only the log file directory that you specify is monitored.
     File Encoding	Select the encoding format of log files.
     First Collection Size	Specify the size of data that Logtail can collect from a log file the first time it does so. Default value: 1024. Unit: KB. If it's less than 1,024 KB, Logtail collects data from the beginning of the file. If it's greater than 1,024 KB, Logtail collects the last 1,024 KB of data in the file. You can configure First Collection Size based on your business requirements. Valid values: 0 to 10485760. Unit: KB.
     Collection Blacklist	If you enable this, configure a blacklist to specify the directories or files that you want Simple Log Service to skip when it collects logs. You can specify exact directories and file names. You can also use wildcard characters to specify directories and file names. When you configure this parameter, you can use only the asterisk (*) or question mark (?) as wildcard characters. Important If you use wildcard characters to specify a value for File Path and you want to skip some subdirectories in the specified directory, configure Collection Blacklist to specify the subdirectories. You must specify complete ones. For example, if you set File Path to /home/admin/app*/log/*.log and you want to skip all subdirectories in the /home/admin/app1* directory, select Directory Blacklist and enter /home/admin/app1*/** in the Directory Name field. If you enter /home/admin/app1*, the blacklist does not take effect. When a blacklist is in use, computational overhead is generated. We recommend a maximum of 10 entries per blacklist. You cannot specify a directory that ends with a forward slash (/). For example, if you specify the /home/admin/dir1/ directory, the directory blacklist does not take effect. The following types of blacklists are supported: File Path Blacklist If you select File Path Blacklist and enter /home/admin/private*.log in the File Path Name field, all files prefixed by private and suffixed by .log in the /home/admin/ directory are skipped. If you select File Path Blacklist and enter /home/admin/private*/*_inner.log in the File Path Name field, all files suffixed by _inner.log in the subdirectories prefixed by private in the /home/admin/ directory are skipped. For example, the /home/admin/private/app_inner.log file is skipped, but the /home/admin/private/app.log file is not. File Blacklist If you select File Blacklist and enter app_inner.log in the File Name field, all files whose names are app_inner.log are skipped. Directory Blacklist If you select Directory Blacklist and enter /home/admin/dir1 in the Directory Name field, all files in the /home/admin/dir1 directory are skipped. If you select Directory Blacklist and enter /home/admin/dir* in the Directory Name field, all files in the subdirectories prefixed by dir in the /home/admin/ directory are skipped. If you select Directory Blacklist and enter /home/admin/*/dir in the Directory Name field, all files in the dir subdirectory in each second-level subdirectory of the /home/admin/ directory are skipped. For example, the files in the /home/admin/a/dir directory are skipped, but those in the /home/admin/a/b/dir directory are not.
     Allow File to Be Collected Multiple Times	By default, you can use only one Logtail configuration to collect logs from a log file. If you want to collect multiple copies of logs from a log file, turn on Allow File to Be Collected Multiple Times.
     Advanced Parameters	Optional. Configure the advanced parameters that are related to input processors. For more information, see CreateLogtailPipelineConfig.

   • Processor Configurations

     Parameter	Description
     Log Sample	Add a sample log collected from an actual scenario. You can use the sample log to easily configure parameters related to log processing. You can add multiple sample logs. Ensure that the total length of them does not exceed 1,500 characters. [2023-10-01T10:30:01,000] [INFO] java.lang.Exception: exception happened at TestPrintStackTrace.f(TestPrintStackTrace.java:3) at TestPrintStackTrace.g(TestPrintStackTrace.java:7) at TestPrintStackTrace.main(TestPrintStackTrace.java:16)
     Multi-line Mode	Specify the type of multi-line logs. A multi-line log spans multiple consecutive lines. You can configure this parameter to identify each multi-line log in a log file. Custom: A multi-line log is identified based on the value of Regex to Match First Line. Multi-line JSON: Each JSON object is expanded into multiple lines. Example: { "name": "John Doe", "age": 30, "address": { "city": "New York", "country": "USA" } } Configure Processing Method If Splitting Fails. Exception in thread "main" java.lang.NullPointerException at com.example.MyClass.methodA(MyClass.java:12) at com.example.MyClass.methodB(MyClass.java:34) at com.example.MyClass.main(MyClass.java:½0) For the preceding sample log, Simple Log Service can discard the log or retain each single line as a log if it fails to split it. Discard: The log is discarded. Retain Single Line: Each line of log text is retained as a log. A total of four logs are retained.
     Processing Method	Add processors as needed. You can add native and extended processors for data processing. Important Refer to the console page prompts for usage restrictions on the processors. Logtail V2.0 You can arbitrarily combine native processors for data processing. You can combine native and extended processors. Extended processors must follow native processors in the sequence. Logtail earlier than V2.0 You cannot add native and extended processors at the same time. You can use native processors only to collect text logs. When you add them, note the following: You must first add one of the following Logtail processors: Data Parsing (Regex Mode), Data Parsing (Delimiter Mode), Data Parsing (JSON Mode), Data Parsing (NGINX Mode), Data Parsing (Apache Mode), and Data Parsing (IIS Mode). After you add the first processor, you can add a Time Parsing processor, a Data Filtering processor, and multiple Data Masking processors. When you configure the Retain Original Field if Parsing Fails and Retain Original Field if Parsing Succeeds parameters, you can use only the following parameter combinations. For others, Simple Log Service does not ensure configuration effects. Upload logs that are parsed. Upload logs that are obtained after successful parsing, and raw ones if the parsing fails. Upload logs obtained after parsing. Add a raw log field to the logs if the parsing succeeds, and raw logs if it fails. For example, if a raw log is "content": "{"request_method":"GET", "request_time":"200"}" and it's successfully parsed, the system adds a raw log field, which is specified by the New Name of Original Field parameter. If you do not configure the parameter, the original field name is used. The field value is {"request_method":"GET", "request_time":"200"}.

6. Create indexes and preview data. Then, click Next. By default, full-text indexing is enabled in Simple Log Service. You can also manually create field indexes for the collected logs or click Automatic Index Generation. Then, Simple Log Service generates field indexes. For more information, see Create indexes.

   Important

   If you want to query all fields in logs, we recommend that you use full-text indexes. If you want to query only specific fields, we recommend that you use field indexes. This helps reduce index traffic. If you want to analyze fields, you must create field indexes. You must include a SELECT statement in your query statement for analysis.

7. Click Query Log. Then, you are redirected to the query and analysis page of your Logstore.

   You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information about how to query and analyze logs, see Guide to log query and analysis.

References

• If the preview page is blank or the query page displays no data when you use Logtail to collect logs, you can troubleshoot the issue by following the instructions provided in What do I do if errors occur when I use Logtail to collect logs? If you use Logtail to collect logs, errors may occur. For example, regular expressions may fail to be parsed, invalid file paths may exist, and traffic may exceed the processing capabilities of shards. You can view Logtail collection errors by following the instructions provided in How do I view Logtail collection errors? For more information about the common errors in data collection, see How do I troubleshoot the common errors that may occur when Simple Log Service collects logs?.

• By default, you can use only one Logtail configuration to collect logs from a log file. If you want to collect multiple copies of logs in a file, you can follow the instructions provided in How do I collect multiple copies of logs in a file?

• For more information about how to collect logs from servers in a corporate intranet to Simple Log Service, see Collect logs from servers in a corporate intranet.