add argparse for long recording/not skipping silence

small clean up
This commit is contained in:
BotoX 2023-01-19 20:21:24 +01:00
parent d4a0d51bfc
commit 9423d2487d
17 changed files with 1753 additions and 284 deletions

View File

@ -1,8 +0,0 @@
language: cpp
os: osx
script:
- g++ -v
- clang++ -v
- cd premake && ./gmake.sh
- cd gmake && make config=release_x32

112
README.md
View File

@ -1,108 +1,10 @@
# demboyz # demboyz
we dem boyz we dem boyz
## Build Status ## Build
| System | Compiler | Status | ```
| ------ | -------- | ------ | cd premake/
| Linux 64-bit | G++-4.8.4 | [![Build Status](https://semaphoreci.com/api/v1/sizzlingcalamari/demboyz/branches/master/badge.svg)](https://semaphoreci.com/sizzlingcalamari/demboyz) | premake5 gmake
| Windows | VS2017 | [![Build status](https://ci.appveyor.com/api/projects/status/pc63pbl9b0t5tygl/branch/master?svg=true)](https://ci.appveyor.com/project/SizzlingCalamari/demboyz/branch/master) | cd gmake
| OSX | Apple LLVM version 7.3.0 (clang-703.0.31) | [![Build Status](https://travis-ci.org/SizzlingStats/demboyz.svg?branch=master)](https://travis-ci.org/SizzlingStats/demboyz) | make -j
```
## What is it?
Demboyz is a command line tool to convert TF2 STV demos into a human readable json format. Demboyz runs on Linux, Windows, and OSX.
It supports the following conversions:
.dem/.json <-> .dem/.json
.dem/.json -> .con
The .dem format is the TF2 STV demo format.
The .json format is the demboyz streaming json demo format.
The .con format is a log equal to what would be produced by TF2
with netmessage/demmessage logs enabled.
## Potential Uses
* Exporting Stats: Parsing for player stats and motion throughout maps.
* Anticheat: Interpreting player net data for malicious behavior and cheating
* Anti-Anticheat: Modifying player net data to hide your malicious behaviour and cheating
* Censoring: Cover up rude communication by removing player voice and chat messages
* Social engineering: Make your friends seem rude by adding phony player voice and chat messages
* ConeBone69 revival: Change the names of all players to ConeBone69
## Usage
./demboyz mystvdemo.dem mystvdemo.json
./demboyz mystvdemo.dem mystvdemo.con
./demboyz mystvdemo.json mystvdemo.dem
./demboyz mystvdemo.json mystvdemo.con
In the first example, mystvdemo.dem will be read from the current working directory, while mystvdemo.json will be written to the current working directory.
## Contributing
### For Developers
If you would like to contribute to demboyz, here are a few tasks up for grabs:
#### General programming experience:
* Add automated tests to demboyz.
* Using the Catch C++ testing framework.
* Unit test the serialization of network messages.
* System test exact binary matches of dem -> json -> dem transformations.
* Document the .dem format.
* Turn the working demboyz serialization code into a reference manual for the .dem fomat.
* Any amount of work on this helps. Document one struct!
* Develop an app that uses demboyz.
* Provide feedback on the usefulness of the json demo format.
* Create new tools previously impossible to make.
#### C++ and asm experience:
* Continue reverse engineering the STV demo structs and serialization (svc_ messages).
* Many of the reversed svc_ messages still have unknown binary chunks of data.
* Reverse them for addition to the demboyz source.
* Reverse engineer the POV demo structs and serialization (clc_ messages).
* Currently, demboyz only supports STV demos due to POV demo messages being separate and unknown.
* Work on reverse engineering the clc_ messages just like the svc_ messages.
* Add a conversion to the replay demo format.
* The replay demo format would require reverse engineering just like POV and STV demos.
### For Users
* Cheer on the developers.
* Pressure the developers.
* Become a developer.
## Who Uses demboyz?
* [KZMod Demo Player [beta]](http://xtreme-jumps.eu/e107_plugins/forum/forum_viewtopic.php?359435) by kraster
## Compiling From Source
When following instructions below, the compiled binary will be output in the bin folder.
### Using Vagrant
# Launch VM and build
vagrant up
# Patch VM and re-build
vagrant provision
# Manual build
vagrant ssh
cd /vagrant/premake
./gmake.sh && cd gmake
make
# Disconnect session and stop VM
exit
vagrant halt
### With Visual Studio 2013
# Generate vs 2013 project
cd premake
vs2013.bat
Open generated VS solution at premake/vs2013/demboyz.sln

125
Vagrantfile vendored
View File

@ -1,125 +0,0 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
# All Vagrant configuration is done here. The most common configuration
# options are documented and commented below. For a complete reference,
# please see the online documentation at vagrantup.com.
# Every Vagrant virtual environment requires a box to build off of.
config.vm.box = "ubuntu/trusty64"
# Disable automatic box update checking. If you disable this, then
# boxes will only be checked for updates when the user runs
# `vagrant box outdated`. This is not recommended.
# config.vm.box_check_update = false
# Create a forwarded port mapping which allows access to a specific port
# within the machine from a port on the host machine. In the example below,
# accessing "localhost:8080" will access port 80 on the guest machine.
# config.vm.network "forwarded_port", guest: 80, host: 8080
# Create a private network, which allows host-only access to the machine
# using a specific IP.
# config.vm.network "private_network", ip: "192.168.33.10"
# Create a public network, which generally matched to bridged network.
# Bridged networks make the machine appear as another physical device on
# your network.
# config.vm.network "public_network"
# If true, then any SSH connections made will enable agent forwarding.
# Default value: false
# config.ssh.forward_agent = true
# Share an additional folder to the guest VM. The first argument is
# the path on the host to the actual folder. The second argument is
# the path on the guest to mount the folder. And the optional third
# argument is a set of non-required options.
# config.vm.synced_folder "../data", "/vagrant_data"
# Provider-specific configuration so you can fine-tune various
# backing providers for Vagrant. These expose provider-specific options.
# Example for VirtualBox:
#
# config.vm.provider "virtualbox" do |vb|
# # Don't boot with headless mode
# vb.gui = true
#
# # Use VBoxManage to customize the VM. For example to change memory:
# vb.customize ["modifyvm", :id, "--memory", "1024"]
# end
#
# View the documentation for the provider you're using for more
# information on available options.
# Enable provisioning with CFEngine. CFEngine Community packages are
# automatically installed. For example, configure the host as a
# policy server and optionally a policy file to run:
#
# config.vm.provision "cfengine" do |cf|
# cf.am_policy_hub = true
# # cf.run_file = "motd.cf"
# end
#
# You can also configure and bootstrap a client to an existing
# policy server:
#
# config.vm.provision "cfengine" do |cf|
# cf.policy_server_address = "10.0.2.15"
# end
# Enable provisioning with Puppet stand alone. Puppet manifests
# are contained in a directory path relative to this Vagrantfile.
# You will need to create the manifests directory and a manifest in
# the file default.pp in the manifests_path directory.
#
# config.vm.provision "puppet" do |puppet|
# puppet.manifests_path = "manifests"
# puppet.manifest_file = "default.pp"
# end
# Enable provisioning with chef solo, specifying a cookbooks path, roles
# path, and data_bags path (all relative to this Vagrantfile), and adding
# some recipes and/or roles.
#
# config.vm.provision "chef_solo" do |chef|
# chef.cookbooks_path = "../my-recipes/cookbooks"
# chef.roles_path = "../my-recipes/roles"
# chef.data_bags_path = "../my-recipes/data_bags"
# chef.add_recipe "mysql"
# chef.add_role "web"
#
# # You may also specify custom JSON attributes:
# chef.json = { mysql_password: "foo" }
# end
# Enable provisioning with chef server, specifying the chef server URL,
# and the path to the validation key (relative to this Vagrantfile).
#
# The Opscode Platform uses HTTPS. Substitute your organization for
# ORGNAME in the URL and validation key.
#
# If you have your own Chef Server, use the appropriate URL, which may be
# HTTP instead of HTTPS depending on your configuration. Also change the
# validation key to validation.pem.
#
# config.vm.provision "chef_client" do |chef|
# chef.chef_server_url = "https://api.opscode.com/organizations/ORGNAME"
# chef.validation_key_path = "ORGNAME-validator.pem"
# end
#
# If you're using the Opscode platform, your validator client is
# ORGNAME-validator, replacing ORGNAME with your organization name.
#
# If you have your own Chef Server, the default validation client name is
# chef-validator, unless you changed the configuration.
#
# chef.validation_client_name = "ORGNAME-validator"
# Shell provisioner
config.vm.provision "shell", path: "vagrant.sh"
end

1686
demboyz/base/argparse.hpp Normal file

File diff suppressed because it is too large Load Diff

View File

@ -1,6 +1,7 @@
#include "io/demoreader.h" #include "io/demoreader.h"
#include "game/sourcecontext.h" #include "game/sourcecontext.h"
#include <base/argparse.hpp>
#include <cstdio> #include <cstdio>
#include <string> #include <string>
#include <cassert> #include <cassert>
@ -8,34 +9,57 @@
int main(const int argc, const char* argv[]) int main(const int argc, const char* argv[])
{ {
if (argc < 2)
{
fprintf(stderr, "Usage: %s <in>.dem [in2.dem] ...\n", argv[0]);
return -1;
}
DemoReader::Init(); DemoReader::Init();
SourceGameContext *context = nullptr; SourceGameContext *context = nullptr;
argparse::ArgumentParser program("demboyz", "1.0", argparse::default_arguments::help);
program.add_argument("-l", "--long")
.default_value(false)
.implicit_value(true)
.help("Don't skip silence.");
program.add_argument("files")
.help("Demo files [.dem]")
.nargs(argparse::nargs_pattern::at_least_one);
program.add_description("Multiple demo files are only supported for split demos, supply in correct order.");
try {
program.parse_args(argc, argv);
}
catch (const std::runtime_error& err) {
std::cerr << err.what() << std::endl;
std::cerr << program;
return -1;
}
bool bSkipSilence = !program.get<bool>("--long");
std::vector<std::string> files;
try {
files = program.get<std::vector<std::string>>("files");
} catch (std::logic_error& e) {
std::cerr << "ERROR: No demo files provided!" << std::endl << std::endl;
std::cerr << program;
return -1;
}
bool error = false; bool error = false;
for (int i = 1; i < argc; i++) for (unsigned int i = 0; i < files.size(); i++)
{ {
std::filesystem::path inputFile(argv[i]); std::filesystem::path inputFile(files[i]);
FILE* inputFp = fopen(inputFile.c_str(), "rb"); FILE* inputFp = fopen((char *)inputFile.c_str(), "rb");
if (!inputFp) if (!inputFp)
{ {
fprintf(stderr, "Error: Could not open input file\n"); fprintf(stderr, "ERROR: Could not open input file '%s'\n", inputFile.c_str());
return -1; return -1;
} }
static std::filesystem::path outputDir = inputFile.filename().replace_extension(); static std::filesystem::path outputDir = inputFile.filename().replace_extension();
static std::filesystem::path outputDirVoice = outputDir.string() + "/voice"; static std::filesystem::path outputDirVoice = outputDir.string() + "/voice";
if (i == 1) if (i == 0)
{ {
std::filesystem::create_directory(outputDir); std::filesystem::create_directory(outputDir);
std::filesystem::create_directory(outputDirVoice); std::filesystem::create_directory(outputDirVoice);
context = new SourceGameContext(outputDir, outputDirVoice); context = new SourceGameContext(outputDir.string(), outputDirVoice.string(), bSkipSilence);
if(!context->init()) if(!context->init())
return -1; return -1;
} }

View File

@ -78,6 +78,7 @@ void Logic::End()
data["voice"]["total_time"] = voiceTotalTime; data["voice"]["total_time"] = voiceTotalTime;
data["voice"]["active_time"] = voiceActiveTime; data["voice"]["active_time"] = voiceActiveTime;
data["voice"]["silence"] = json::array();
for(const auto& o : context->voiceWriter->m_silence) for(const auto& o : context->voiceWriter->m_silence)
{ {
data["voice"]["silence"] += json({o.first, o.second}); data["voice"]["silence"] += json({o.first, o.second});

View File

@ -15,9 +15,10 @@
#include "netmessages/svc_voiceinit.h" #include "netmessages/svc_voiceinit.h"
#include "netmessages/svc_voicedata.h" #include "netmessages/svc_voicedata.h"
SourceGameContext::SourceGameContext(std::string outputDir, std::string outputDirVoice): SourceGameContext::SourceGameContext(std::string outputDir, std::string outputDirVoice, bool bSkipSilence):
outputDir(outputDir), outputDir(outputDir),
outputDirVoice(outputDirVoice) outputDirVoice(outputDirVoice),
m_bSkipSilence(bSkipSilence)
{ {
stringTables = new StringTableContainer(this); stringTables = new StringTableContainer(this);
userIdLookUp = new uint8_t[USHRT_MAX+1]; userIdLookUp = new uint8_t[USHRT_MAX+1];
@ -51,7 +52,7 @@ bool SourceGameContext::init()
return false; return false;
} }
voiceWriter = new VoiceDataWriter(this, outputDirVoice.c_str()); voiceWriter = new VoiceDataWriter(this, outputDirVoice.c_str(), m_bSkipSilence);
if(!voiceWriter->init()) if(!voiceWriter->init())
return false; return false;

View File

@ -69,7 +69,7 @@ typedef struct player_info_s
struct SourceGameContext struct SourceGameContext
{ {
SourceGameContext(std::string outputDir, std::string outputDirVoice); SourceGameContext(std::string outputDir, std::string outputDirVoice, bool bSkipSilence);
~SourceGameContext(); ~SourceGameContext();
bool init(); bool init();
@ -88,6 +88,7 @@ struct SourceGameContext
std::string outputDir; std::string outputDir;
std::string outputDirVoice; std::string outputDirVoice;
bool m_bSkipSilence;
FILE* outputFp; FILE* outputFp;
Logic* logic; Logic* logic;

View File

@ -132,9 +132,10 @@ int SilkVoiceDecoder::Decompress(
} }
VoiceDataWriter::VoiceDataWriter(SourceGameContext* context, const char* outputPath): VoiceDataWriter::VoiceDataWriter(SourceGameContext* context, const char* outputPath, bool bSkipSilence):
context(context), context(context),
m_outputPath(outputPath) m_outputPath(outputPath),
m_bSkipSilence(bSkipSilence)
{ {
} }
@ -195,7 +196,7 @@ void VoiceDataWriter::EndCommandPacket(const PacketTrailingBits& trailingBits)
return; return;
// Skip silence if noone talks for at least 1.5 seconds // Skip silence if noone talks for at least 1.5 seconds
if((m_curTick - m_lastVoiceTick) / context->fTickRate > 1.5) if((m_curTick - m_lastVoiceTick) / context->fTickRate > 1.5 && m_bSkipSilence)
{ {
if(!m_isSilenced) if(!m_isSilenced)
{ {

View File

@ -53,7 +53,7 @@ private:
class VoiceDataWriter class VoiceDataWriter
{ {
public: public:
VoiceDataWriter(SourceGameContext *context, const char* outputPath); VoiceDataWriter(SourceGameContext *context, const char* outputPath, bool bSkipSilence);
bool init(); bool init();
void Start(); void Start();
@ -90,6 +90,7 @@ private:
int32_t m_silenceTicks = 0; int32_t m_silenceTicks = 0;
int32_t m_silenceTicksStart = 0; int32_t m_silenceTicksStart = 0;
const char* m_outputPath = nullptr; const char* m_outputPath = nullptr;
bool m_bSkipSilence = true;
int16_t m_decodeBuffer[32768]; int16_t m_decodeBuffer[32768];

View File

@ -1,13 +0,0 @@
#!/bin/bash
if [ "$(uname)" == "Darwin" ]; then
# Do something under Mac OS X platform
./premake5_osx gmake --file=premake5.lua
elif [ "$(expr substr $(uname -s) 1 5)" == "Linux" ]; then
# Do something under Linux platform
./premake5_linux gmake --file=premake5.lua
elif [ "$(expr substr $(uname -s) 1 10)" == "MINGW32_NT" ]; then
# Do something under Windows NT platform
echo "windows? don't use this"
fi

Binary file not shown.

View File

@ -1,25 +1,26 @@
solution "demboyz" solution "demboyz"
basedir ".." basedir ".."
location (_ACTION) location (_ACTION)
targetdir "../bin" targetdir "../bin"
startproject "demboyz" startproject "demboyz"
configurations { "Release", "Debug" } configurations { "Release", "Debug" }
flags { "MultiProcessorCompile", "Symbols" } flags { "MultiProcessorCompile" }
symbols "On"
configuration "Debug" filter "configurations:Debug"
defines { "DEBUG" } defines { "DEBUG" }
configuration "Release" filter "configurations:Release"
optimize "Full" optimize "Full"
configuration {} filter {}
project "demboyz" project "demboyz"
kind "ConsoleApp" kind "ConsoleApp"
language "C++" language "C++"
configuration "gmake" filter "configurations:gmake"
buildoptions { "-std=c++17 -Wall" } buildoptions { "-std=c++17 -Wall" }
linkoptions { "-flto -no-pie -Wall" } linkoptions { "-flto -no-pie -Wall" }
configuration {} filter {}
files files
{ {
"../demboyz/**.h", "../demboyz/**.h",

Binary file not shown.

Binary file not shown.

View File

@ -7,9 +7,10 @@ group "external"
kind "StaticLib" kind "StaticLib"
language "C++" language "C++"
location (_ACTION .. "/" .. project().name) location (_ACTION .. "/" .. project().name)
configuration "gmake" filter "configurations:gmake"
buildoptions { "-std=c++11" } buildoptions { "-std=c++11" }
configuration {} filter {}
includedirs includedirs
{ {
base_dir .. "include/", base_dir .. "include/",

View File

@ -1,4 +0,0 @@
#!/bin/sh
# Install g++ dependencies
sudo apt-get install -y build-essential g++-4.8 g++-4.8-multilib